Voice controlled assistance for monitoring adverse events of a user and/or coordinating emergency actions such as caregiver communication

ABSTRACT

A user, such as an elderly person, may be assisted by an assistance device in his or her caregiving environment that operates in conjunction with one or more server computers. The assistance device may execute a schedule of assistance actions where each assistance action is associated with a time and is executed at that time to assist the user. An assistance action may present an input request to a user, process a voice input of the user, and analyze the voice input to determine that the voice input corresponds to a negative response event, a positive response event, or a non-response event. Based on the categorization of one or more voice inputs as negative response events, positive response events, or non-response events, it may be determined to notify a caregiver of the user, for example where the user has not responded to a number of assistance actions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/590,012, filed on May 9, 2017 and entitled “VOICE CONTROLLEDASSISTANCE FOR MONITORING ADVERSE EVENTS OF A USER AND/OR COORDINATINGEMERGENCY ACTIONS SUCH AS CAREGIVER COMMUNICATION”, the contents ofwhich are incorporated herein by reference.

FIELD OF TECHNOLOGY

This disclosure relates generally to data processing devices and, moreparticularly, to a method, a device, a system and/or a manufacture ofvoice controlled assistance for monitoring adverse events of a userand/or coordinating emergency actions such as caregiver communication.

BACKGROUND

A caregiving organization that provides care to a user within acaregiving environment may need to monitor the user for wellbeing,health, and/or for the safety of the user in case of an incident. Forexample, a user who is impaired may be unable get up if they fall down,or may be unable to escape a house if the house catches on fire.Similarly, the user may suffer from intermittent memory loss that may becorrelated with an increase risk of incident. It may be difficult forthe caregiving organization (e.g., a hospital, a hospice care, a in-homehealthcare organization), a caregiver (e.g., a hospice worker, a nurse,a family member) providing care to determine how the user is doingand/or a current risk level of the user, including but not limited to ina home care environment. If the user is subject to an incident or beginsto undergo a temporary or permanent mental change, the user may havedifficulty getting to a communication device (e.g., a phone, a mobiledevice, a computer, a panic button on a necklace that dials 9-1-1) tocontact the caregiver or an emergency service. The caregiver and/or thecaregiving organization may therefore have little insight in the user'sdaily state. As a result, the user may be at increased risk of injury oreven death.

At the same time, a voice controlled automated assistance service suchas Google Assistant, Amazon Alexa, and Apple Siri may provide a generalassistance that may be useful to any user, but may be especially helpfulfor persons who are impaired, elderly, under treatment in a medicalfacility, or undergoing home care. For example, the user may more easilyinterface with technology by using their voice which may be a naturalway of communicating for the user. Such services may continue to becomeincreasingly human-like in their responses and interactions. The usermay therefore be comforted by voice communications or find companionshipwith the automated voice assistance service. For example, the user maybe able to search the web, order a product from an online retailer, andcontrol household connected devices such as a smart bulb or a connectedtelevision. They may therefore by encouraged to interact frequently,provided candid or honest information, and engage with and confide inthe automated assistance service.

However, utilizing voice controlled automated assistances services maypresent some difficulties for a user. Assistance applications of theautomated assistance service may have a complicated set of voice inputsor commands that the user may have difficulty remembering orpronouncing. The automated assistance service may have a limited meansof customization, including but not limited to voice commands that mayinvoke assistance applications and initiate assistance actions such asordering delivery services. Some of these difficulties may be amplifiedin impaired persons, medical patients and/or the elderly. In addition,the user (and/or the caregiver) may misunderstand a capability of theautomated assistance service as it relates to safety of the user. Forexample, the automated assistance service may be unable to distinguishwhen a user is in peril versus trying to utilize a general service. Theautomated assistance service may have no programming to support anemergency response to the incident or call for help by the user.

As a result of these challenges, the caregiving organization and/or thecaregiver may not know the user is at increased risk for an incident orthat the user is in immediate danger. The user may continue to besubject to a high risk of injury, immediate risk of injury, and/or evenrisk of death, including as a result of misunderstanding thecapabilities of the voice controlled automated assistance service. Otherdevices meant to aid the user if an incident occurs may be unavailable,require physical possession, and/or be unusable during an emergency dueto the user's infrequent use of such devices.

SUMMARY

Disclosed are a method, a device, a system, and/or a manufacture ofvoice controlled assistance for monitoring adverse events of a userand/or coordinating emergency actions such as caregiver communication.

In one embodiment, an apparatus includes a speaker for generating averbal status query to a user and a microphone to gather from animmediate environment of the microphone a voice communication of a userand/or an ambient sound of the immediate environment. The apparatusfurther includes a network interface controller providing communicativecoupling to at least one instance of a voice controlled assistanceservice and at least one instance of an automated emergency assistanceservice. The apparatus includes a processor and a memory. The memoryincludes computer-executable instructions that when executed on theprocessor cause the processor to: (i) generate the verbal status queryfor the user; (ii) determine occurrence of an event of concern includinga non-response event defined as a failure to receive by the microphone arecognizable response from the user; (iii) determine that the event ofconcern is an adverse event requiring transmission of a notification toa device of a caregiver of the user, and (iv) initiate an alert messagenotification to the device of the caregiver of the user based on theadverse event.

The apparatus may further include a presence sensor, and the memory mayfurther include computer-executable instructions that when executed onthe processor cause the processor to generate the voice status query forthe user in response to a sensing event of the presence sensor sensing apresence of the user. The apparatus may include a database to store asafety profile that may specify a condition under which one or moreevents of concern define the adverse event requiring notification to thedevice of the caregiver of the user. The memory may further include anassistance service coordinator that includes computer-executableinstructions that when executed on the processor cause the processor to,upon determination of the adverse event, automatically re-route adestination of a set of voice communications of the user from a firstvoice controlled assistance service (that provides a general assistance)to a second voice controlled assistance service providing an automatedemergency assistance service to the user.

The apparatus may further include within the memory computer-executableinstructions that when executed on the processor cause the processor torepeat the verbal status query to the user and increase a volume of theverbal status query from the speaker following determination of thenon-response event of the user. The computer-executable instructions mayalso, upon occurrence of the event of concern, transmit a status queryto a device of the user and/or generate a verbal warning to the userwith the speaker. The verbal warning may state that (i) the user hasfailed to provide the recognizable response, (ii) the caregiver has beenalerted, and/or (iii) the emergency service has been alerted. Thecomputer executable instructions may also generate the voice query withthe speaker upon detection of a sound spike and/or a vibration spike(e.g., that may indicate that the user has fallen or collided with anobject).

The apparatus may further include a particulate sensor to detect withinthe environment of the user a particulate associated with a first healthrisk to the user. Similarly, the apparatus may include a chemical sensorto detect within the environment of the user a chemical associated witha second health risk to the user. The memory may includecomputer-executable instructions that when executed on the processorcause the processor to detect the particulate at the particulate sensorand/or the chemical at the chemical sensor, and determine an adverseevent requiring execution of an emergency action based on the detectionof the particulate at the particulate sensor and/or the chemical at thechemical sensor. The chemical sensor may be a carbon dioxide sensorand/or a carbon monoxide sensor, and the particulate sensor may be asmoke detector.

The apparatus may further include a user interface, to display a textualmessage converted from a speech response of the voice controlledassistance server. The memory may further include a speech-textconverter including computer-executable instructions that when executedon the processor cause the processor to convert the speech response ofthe assistance service to the textual message for display on a graphicaluser interface. The apparatus may also include a video camera, thememory further including computer-executable instructions that whenexecuted on the processor cause the processor to, upon occurrence of theadverse event, establish a video feed between the assistance device andthe emergency service and/or the device of the caregiver. The computerreadable instructions may also populate the graphical user interfacewith the textual message converted with the speech-text converter forpresentation to the user. The computer executable instructions maydetermine a threshold inactivity of the device of the user, and thepresence sensor may be a motion sensor, an infrared sensor, a proximitysensor, an RFID sensor, a noise sensor, a Bluetooth sensor, and/or aultrasonic frequency sound sensor.

The event of concern may further include a negative response event ofthe user defined as a negative response from the user to the verbalstatus query. The threshold inactivity may be determined based on a GPScoordinate of the device, a network traffic of the device of the user,an accelerometer data of the device of the user, a microphone of thedevice of the user, and/or a video feed of the device of the user.

The memory may include an assistance service scheduler that includescomputer-executable instructions that when executed on the processorcause the processor to define a custom command word and associate thecustom command word with an assistance application of a voice controlledassistance service along with an assistance action of the assistanceapplication. The assistance action may conform to a voice commandprotocol of the assistance application. The memory may pre-specify oneor more voice inputs required upon initiation of the assistance actionto execute the assistance action of the assistance application, the oneor more voice inputs conforming to the voice command protocol of theassistance application. The computer executable instructions may alsospecify: (i) a confirmation requirement for a verbal confirmation of theuser prior to execution of the assistance application and/or (ii) aninput request for one or more additional verbal inputs conforming thevoice command protocol before execution of the assistance application.The computer executable instructions stored in the memory may scheduleexecution of the assistance action of the assistance application. Theevent of concern may further include failure to receive the verbalconfirmation of the user and/or a failure to receive the input requestfor one or more additional verbal inputs.

In another embodiment, a system includes an assistance devicecommunicatively coupled to a voice controlled assistance service over anetwork. The assistance device includes a speaker of the assistancedevice, a microphone of the assistance device, a processor of theassistance device, a presence sensor of the assistance device (to sensea presence of a user of the assistance device within an immediateenvironment of the assistance device), and a memory of the assistancedevice. The memory includes computer-executable instructions that whenexecuted on the processor cause the processor to, upon sensing thepresence of the user, generate a verbal status query to the userincluding an authorization request for execution of an assistance actionof an assistance application of a voice controlled assistance service.The assistance action conforms to a voice command protocol of theassistance application. The computer executable instructions furtherinvoke the assistance application and initiate the assistance actionupon receiving an authorization from the user.

The system further includes a sensing pod to expand communicationbetween the user and the assistance device to an extended environment,the sensing pod including a microphone of the sensing pod to gather asound signal from an immediate environment of the sensing pod that maybe determined by the assistance device to be a non-response event of theuser to the verbal status query and/or a negative response event of theuser defined as a negative response from the user to the verbal statusquery. The sensing pod further includes a speaker of the sensing pod(optionally to generate the verbal status query), a processor of thesensing pod, a memory of the sensing pod and a wirelesstransmitter-receiver for communicative coupling with the assistancedevice over the network to transmit the sound signal. The system mayfurther include a mobile device, a wearable device, including, anautomated assistance server (e.g., for providing a general voiceassistance of the voice controlled assistance service), an assistancescheduling server (e.g., for streamlining interactions with theautomated assistance service and its assistance applications), and anemergency server (e.g., for providing a specialized emergencyassistance), and additional elements of each.

In yet another embodiment, a method includes generating with a speaker averbal status query to a user at a predetermined time, at a random time,upon a sensing event in which a presence of the user is sensed, and/orupon a threshold inactivity of a device of the user. A countdown timerset to a time length is initiated. The method determines occurrence ofan event of concern that is (i) a negative response event that is anegative response from the user to the verbal status query, the negativeresponse received by a microphone, and/or (ii) a non-response event thatis a failure to receive by the microphone a recognizable response fromthe user before expiration of the countdown timer, The method referencesa safety profile specifying a condition under which one or more eventsof concern define an adverse event requiring execution of an emergencyaction. The method exterminates occurrence of the adverse event based onthe condition of the safety profile and occurrence of the event ofconcern. The method determines the emergency action requires anotification to a device of a caregiver of the user and/or a device ofan emergency service. An alert message is generated for a device of theemergency service and/or the device of the caregiver of the user, thealert message including a log including the adverse event and/or the oneor more events of concern.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of this disclosure are illustrated by way of example andnot limitation in the figures of the accompanying drawings, in whichlike references indicate similar elements and in which:

FIG. 1 illustrates an assistance coordination network comprising anassistance device, a sensing pod, an automated assistance server toprovide a general assistance to a user, an emergency server to providean emergency assistance to the user, a scheduling server, and one ormore devices, with one aspect of the assistance coordination networkfurther illustrated in which the assistance device generates anautomated communication (for example, a verbal status query such as“Good morning, how are you?”) to which the user responds with a voicecommunication which may be determined to be an event of concern (forexample, the voice response of “I feel sick”) that may additionally bedetermined to be an adverse event (e.g., exceeding three instances of anegative response event), the adverse event triggering an emergencyaction such as notifying a caregiver of the user, notifying an emergencyservice, and/or re-routing the voice communications of the user from theautomated assistance server to the emergency server, according to one ormore embodiments.

FIG. 2 illustrates the assistance device of FIG. 1, including a presencesensor that may determine a presence of the user in a sensing event toinitiate the verbal status query, a sensing algorithm, a countdowntimer, a status query procedure, an assistance service coordinator, anda database comprising a custom command word, an action instruction tostreamline usage of an assistance application of the automatedassistance service, and a log of responses of the user including but notlimited to a negative response event, a non-response event, and apositive response event, according to one or more embodiments.

FIG. 3 illustrates a sensing pod that can be communicatively coupled tothe assistance device of FIG. 2 to enable an extended environment forwhich the assistance device can perceive and/or communicate with theuser, the sensing pod having its own immediate environment of thesensing pod and further including a capability to generate an audio feedwith a microphone of the sensing pod and/or a video feed with a videocamera of the sensing pod, according to one or more embodiments.

FIG. 4 illustrates the automated assistance server of FIG. 1 forproviding a general assistance to the user (for example, initiating aweb search, ordering a product, engaging in a social interaction, andsimilar functions), the automated assistance server including a speechrecognition engine to recognize and/or translate a speech of the user, apersonal assistance engine for providing the general assistance andgenerating speech responses for the user, and one or more assistanceapplications that can be invoked by an invocation command and that carryout an assistance action initiated by an initiation command, accordingto one or more embodiments.

FIG. 5 illustrates the emergency server of FIG. 1 providing aspecialized emergency assistance to the user (e.g., automaticallyconnecting the user to the emergency service such as 9-1-1, providing anincrease sensitivity to an emergency word, and/or aiding the user indetermining a risk) rather than the general assistance service of theautomated assistance server of FIG. 4, the emergency server including anemergency speech recognition engine to recognize and/or translate anemergency speech of the user (e.g., an emergency word such as “help”and/or a distressed voice tone), an automated assistance engine forproviding the specialized emergency service to the user, and a databaseof the emergency server having the log of the events of concern and/orthe adverse events, according to one or more embodiments.

FIG. 6 illustrates the scheduling server of FIG. 1 including anassistance service scheduler that may associate and store the invocationcommand, the initiation command, and a set of one or more voice inputsof FIG. 4 that may be required for execution of the assistanceapplication, the scheduling server capable of initiating the assistanceaction upon occurrence of an initiation condition to aid in providingthe general assistance to the user and/or for coordination with theemergency server in determining the event of concern in relation to theuser's interaction during the general assistance, according to one ormore embodiments.

FIG. 7 illustrates the device of FIG. 1 of the user, the caregiver,and/or the emergency service, for example a mobile device, a tablet, awearable device (e.g., a smartwatch, an electronic necklace with a panicbutton), a connected household device, and/or a desktop computer, thedevice of FIG. 7 including, for example, an accelerometer, a GPS unit, apresence signaler, a display, a speaker, a microphone, and a videocamera, the speaker and the video camera usable to establishcommunication between and among devices of the user, of the caregiver,and/or of the emergency service as one of the one or more emergencyactions coordinated by the assistance device of FIG. 1, according to oneor more embodiments.

FIG. 8 illustrates a first caregiving environment network in whichaspects of the assistance coordination network of FIG. 1 are deployedwithin and outside of a caregiving environment, a local area network(LAN) communicatively coupling the assistance device of FIG. 2 with twoinstances of the sensing pod of FIG. 3 to provide the extended networkof the assistance device, the user and/or a device of the user sensed ina sensing event of a first sensing pod and an interaction of the usertransmitted to the assistance device and a log comprising theinteraction then transmitted to the mobile device of the caregiving ofthe user through a wide area network (WAN), according to one or moreembodiments.

FIG. 9 illustrates a detailed view in which the interaction of FIG. 8may occur in which the sensing pod determines a presence of the userthrough a sensing event, generates an automated communication such as averbal status query to the user (e.g., “Isn't it a nice day? What do youthink?”), and the user responds with a voice communication that may be apositive response (e.g., “Yes, it's nice out”) or a negative response(e.g., “No”, or “I don't care”), the positive response and/or thenegative response determined by the assistance device and/or theemergency server to be a positive response event or a negative responseevent, respectively, which may be stored as data in the log for possiblerecordkeeping, analysis and/or communication to the caregiver and/or theemergency server, according to one or more embodiments.

FIG. 10 illustrates a second caregiving environment network in which theverbal status query may be generated based on a factor other than thesensing event of FIG. 9, for example upon a threshold inactivity of theuser, the sensing pod then receiving a non-response from its immediateenvironment, expanding the verbal status query and/or a status querythat may be in a different medium to the mobile device of the user,logging a non-response event that may trigger an alert message to themobile device of the caregiver, and/or establishing a video feed and/oran audio feed from the assistance device, the sensing pod, and/or themobile device of the user to determine a state of the user within thecaregiving environment, according to one or more embodiments.

FIG. 11 illustrates a routing of a voice communication of the userbetween the automated assistance server of FIG. 4 providing the generalassistance and the emergency server providing of FIG. 5 providing thespecialized emergency service, a voice communication of the user thatrelates to the general assistance (e.g., “what time is the game on?”)routed to the automated assistance server, whereas the voicecommunications of the user following an event of concern, an adverseevent, and/or upon recitation of certain verbal commands (e.g., “Help”)routed to the emergency server by the assistance coordinator of FIG. 2,according to one or more embodiments.

FIG. 12 illustrates numerous capabilities of the scheduling server ofFIG. 6 to ease interaction with the automated assistance service andprovide an opportunity for determining events of concern, thecapabilities including but not limited to: invocation of an assistanceapplication and/or initiation of an assistance action of the assistanceapplication via a custom command word; initiation of the assistanceaction and/or assistance application upon an initiation condition suchas a predetermined time or upon a sensing event within a timeframe(e.g., asking the user if they would like dinner ordered from a deliveryservice if they are sensed entering a kitchen between 5 PM and 7 PM);reducing verbal inputs required from the user to initiate an assistanceaction; and initiating a confirmation request and/or an authorizationrequest before executing the assistance action and/or the assistanceapplication, according to one or more embodiments.

FIG. 13 is a monitoring and assistance process flow that illustrates oneaspect of the assistance coordination network of FIG. 1 in which astatus query is triggered, a response to the status query is parsed todetermine an event of concern, the event of concern is determined toarise to the level of an adverse event possibly based on previous eventsof concern, and an emergency action is initiated, according to one ormore embodiments.

FIG. 14 is an adverse event determination process flow illustratingadditional aspects of the process flow of FIG. 13, additionallyincluding expansion of a communication environment (e.g., acommunication related to an immediate environment of a sensing podexpanded to an extended environment including all sensing pods,assistance devices, and/or devices of the user within the caregivingenvironment of FIG. 8), logging non-negative responses, and adjustingconditions under which the event of concern and/or adverse event isdetermined, according to one or more embodiments.

FIG. 15 is an emergency action process flow illustrating one or moreemergency actions taken in response to determination of the adverseevent of FIG. 14, including but not limited generating an alert messagefor the caregiver of the user and/or the emergency service, routing avoice communication of the user to the emergency server of FIG. 5,increasing a sensitivity to an emergency speech, establishing an audioand/or video feed to establish communication with the user, and/orexecution of additional emergency actions, according to one or moreembodiments.

FIG. 16 illustrates additional aspects and one specific example of thedetermination of the adverse event, according to one or moreembodiments.

FIG. 17 illustrates additional aspects and one specific example of thedetermination of the emergency action, according to one or moreembodiments.

FIG. 18 is an assistance application automation process flowillustrating selection of an assistance application and an assistanceaction of the assistance application conforming to a voice commandprotocol, pre-storing one or more voice inputs required upon initiationof the assistance application for its execution, determining anyconfirmation requirements and/or voice inputs required from the userjust prior to execution, and setting a condition for triggering theassistance application, according to one or more embodiments.

FIG. 19 is a customizable command process flow illustrating theautomation of the assistance action of FIG. 18 utilizing a customcommand word of the user for the user to initiate the assistanceapplication on demand, according to one or more embodiments.

FIG. 20 is an adverse event calibration process flow illustratingadjustment of the determination of the adverse event requiring anemergency action in response to the determination of an event of concernand/or an adverse event, for example by lowing a threshold fordetermining an event of concern by decreasing a time length of acountdown timer in which the user can respond to the status query beforedetermination of a non-response event, according to one or moreembodiments.

FIG. 21 is a personal vitals and environmental hazard emergency actionprocess flow illustrating detection by the assistance device of FIG. 2and/or the sensing pod of FIG. 3 an adverse vital sign of the userand/or an environmental hazard, the sensing of the personal vital signand/or the environmental hazard (e.g., smoke or carbon monoxide)determined to be an adverse event triggering a warning to the userwithin the caregiving environment, an alert message to the caregiver, analert message to the emergency service, and/or additional emergencyactions, according to one or more embodiments.

FIG. 22 is a user message process flow illustrating a process by whichthe user may store an away message associated with an expectedinactivity time suspending a generation of one or more verbal statusqueries to the user, the away message communicated to the caregiverautomatically and/or upon the non-response event, according to one ormore embodiments.

FIG. 23 is an interaction of concern process flow illustrating a statusquery that may be in the form of a question, a puzzle, a quiz, or atrivia to determine a cognitive ability of the user includingdetermination of an event of concern for incorrect and/or non-responsiveanswers, according to one or more embodiments.

Other features of the present embodiments will be apparent from theaccompanying drawings and from the detailed description that follows.

DETAILED DESCRIPTION

Disclosed are a method, a device, a system and/or a manufacture of voicecontrolled assistance for monitoring adverse events of a user and/orcoordinating emergency actions such as caregiver communication. Althoughthe present embodiments have been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the various embodiments.

FIG. 1 illustrates an assistance coordination network comprising anassistance device 200, a sensing pod 300, an automated assistance server400 to provide a general assistance to a user 100, an emergency server500 to provide an emergency assistance to the user 100, a schedulingserver 600, and one or more devices 700A through 700C for communicationbetween and among the user 100, a caregiver 130, and/or an emergencyservice 150, according to one or more embodiments. In the embodiment ofFIG. 1, one aspect of the assistance coordination network is illustratedin which a user 100 within a caregiving environment (e.g., a house wherethe user 100 is receiving homecare, a nursing facility, a hospital) isfurther within an immediate environment 102A of the assistance device200. The immediate environment 102 of the assistance device 200 is anarea around the assistance device 200 in which the assistance device 200can reliably receive on a microphone and recognize verbal communicationsof a user 100 under a normal range of speech decibels (e.g.,approximately 60 decibels to 90 decibels at a point of origin). Forexample, the immediate environment 102 may be a large living room, aradius of 40 feet around the assistance device 200, two adjacent roomswithin a home, a back yard, etc. The immediate environment 102 maydepend on a microphone sensitivity and/or a voice recognition capabilityof the assistance device 200 and/or voice recognition capability ofservices accessible through the assistance device 200.

In the embodiment of FIG. 1 the user 100 may interact through theassistance device 200 and the sensing pod 300 with an automatedassistance service operating from an automated assistance server 400.The automated assistance service is a voice controlled automatedassistance service that is software and/or computer code that provides ageneral assistance, for example, Apple Siri, Google Assistant, SamsungBixby, and/or Amazon Alexa. The user 100 may be able to use a voicecommand to search the web, find out the weather, order a product from ane-commerce retail store, control a connected device such as a householdthermostat, light, or speaker, and/or engage in similar actions. Speechresponses (e.g., an automated communication 106) are generated by apersonal assistance engine 462 of the automated assistance server 400that may be proprietary and/or open source software that implements“artificial intelligence” interactions and synthetic personalities. Theautomated assistance service 400 may include one or more assistanceapplications 470 that are voice controlled software applicationsaccessible through the automated assistance service (e.g., “AlexaSkills”) and that may provide the user 100 with a specialized service,for example ordering a specific product from a specific vendor (e.g., anapplication or ordering a pizza from a certain pizza company).

While the user 100 in the caregiving environment may engage in generalcommunication and voice interactions with the automated assistanceserver 400, the assistance device 200 may act as an additional interfaceand/or voice controlled interface to the automated assistance serviceimplemented via the automated assistance server 400. As the additionalinterface to the automated assistance server 400, the assistance device200 may provide streamlined commands that may help the user 100 toutilize the assistance applications and assistance actions of theassistance applications (e.g., the assistance application 470 of and theassistance actions 478 of FIG. 4). However, the assistance device 200acting as the additional interface may also provide an integratedmonitoring capability that monitors and/or analyzes the user 100including in the user 100's interactions or attempted interactions withthe automated assistance service, according to one or more embodiments.This may provide frequent and/or accurate data due to the user 100'sfamiliar with user, high level of utility, and engaging socialinteraction.

In the embodiment of FIG. 1, the monitoring may begin upon a sensingevent 104 by the assistance device 200. The sensing event 104 may occurby determining a sound of the user 100 in the immediate environment 102Aof the assistance device 200 or upon sensing the user 100 and/or adevice of the user 100 with a presence sensor 202 utilizing other meansthan sound. The assistance device 200 may then trigger an automatedcommunication 106 from the speaker 204. Specifically, in the embodimentof FIG. 1, the automated communication 106 is a verbal status query 108to the user 100. The verbal status query 108 may be a verbal question orinterrogatory generated as an audio signal and usable to elicit aresponse from the user, the response usable to determine a state, adisposition, a feeling, an emotion, a condition (e.g., physical, mental,psychological, and/or emotional) and/or a status of the user 100. Theuser 100 may reply to the verbal status query 108 with a voicecommunication 110 which is received by a microphone 206 of theassistance device 200. The response of the user 100 may be analyzed anddetermined to be a negative response event (e.g., the negative responseevent 115, not shown in FIG. 1) or a positive response event (e.g., thepositive response event 119, also not shown in FIG. 1), thedetermination made either locally at the assistance device 200, at theemergency server 500, and/or at another location. The negative responseevent 115, the positive response event 119, or a non-response event 117(as shown and described in conjunction with FIG. 10) are data that maybe logged in a database 205 of the assistance device 200. The positiveresponse event 119 may be logged in a log 113 (as shown and described inconjunction with FIG. 2). The log 113 may be periodically provided tothe device 700B of the caregiver 130 in the form of a report, even iflog primarily (or exclusively) comprises instances of the positiveresponse event 119. In contrast, the negative response event 115 and thenon-response event 117 are each instances of an event of concern 116.Occurrence of one or more of the events of concern 116, under suchconditions as may be specified in a safety profile 114 of the user 100,may equate to an adverse event 118 requiring an emergency action 120such as a notification of a caregiver 130 of the user 100.

For example, in the embodiment of FIG. 1 the presence of the user 100within the immediate environment 102A of the assistance device 200 maybe sensed through the sound of the user 100's footsteps above an ambientsound threshold of the immediate environment 102A. The assistance device200 may generate a verbal status query 108 such as “How are you doingtoday?”. In reply, the user 100 may generate a voice communication 110that is a voice response such as “I'm doing ok.” The assistance device200 may generate a second statue query 108 such as “How is your kneefeeling?” based on information previously stored about the user 100 inthe safety profile 114. The user 100 may reply “It hurts.” The firstvoice response of the user 100 may be determined to be a positiveresponse event 119 and the second voice response of the user 100 may bedetermined to be the negative response event 115 that may arise to anevent of concern 116.

The assistance device 200 may then reference the log 113 and the safetyprofile 114. The safety profile 114 comprises data specifying acondition under which one or more events of concern 116 define anadverse event 118 that requires execution of an emergency action 120. Inthe embodiment of FIG. 1, for example, the safety profile 114 of theuser 100 may specify that an adverse event 118 occurs any time the user100 generates: (i) more than three instances of the negative responseevent 117 within a twenty-four hour period; (ii) more than fiveinstances of any event of concern 116 (including instances of thenon-response event 117); or (iii) any negative response event 117related to the user 100's knee or current prescribed medication. Inaddition to a local determination of the event of concern 116, a signalof the voice communication 110 may be communicated through a network 101to the automated assistance server 400 and/or the emergency server 500for the determination.

Upon determination of the adverse event 118, the assistance device 200may determine one or more instances of the emergency action data 121that apply (e.g., by referencing the safety profile 114) and execute anemergency action 120 associated with the emergency action data 121. Inthe embodiment of FIG. 1 the emergency action data 121 specifies analert message 122 to be generated along with the data required forrouting the message. In the embodiment of FIG. 1 the emergency action120 is the specific communication of the alert message 122. The alertmessage 122 includes the log 113 comprising one or more instances of thenegative response event 115 forming the basis of the alert message 122.For example, the log 113 may include the negative response event 115that is a text string of the response of the user 100 to the verbalstatus query 108: “Did you take your medicine this morning?” The user100 may have replied “No, I didn't feel well.”

The alert message 122 is communicated through the network 101 to thedevice 700B of the caregiver 130 and/or the device 700C of the emergencyservice 150. The network 101 may be the internet, a local area network,a wide area network, a mesh network implemented by communicationdevice-to-device, and/or a virtual private network. The device 700Band/or the Device 700C may be a mobile device (e.g., a smartphone, acell phone), a wearable device, a desktop computer, and/or a tabletdevice. The device 700C may also be a server of the emergency service150 that may route, respond to, and/or allocate emergency resources toemergency calls and/or other emergency-related electroniccommunications. The emergency service, for example, may be a policedepartment, a fire department, a hospital, a poison control center,and/or similar agencies and enterprises. In one or more embodiments,where the adverse event 118 may be of lesser concern (e.g., twoconsecutive instances of the negative response event 115) the alertmessage 122 may be transmitted exclusively to the device 700B of thecaregiver 130. In one or more other embodiments, the alert message 122may be of greater concern (e.g., five consecutive instances of thenon-response event 117), and the alert message may be sent to the device700C of the emergency service 150 and/or the device 700B of thecaregiver 130. The result will be to inform the caregiver 130 and/or theemergency 150 that the user 100 may be having difficulty or even may bein imminent peril.

Upon receipt of the alert message 122 the caregiver 130 may be able totake actions on the device 700B such as initiate an audio feed (e.g.,the audio feed 350 of FIG. 3) or a video feed (e.g., the video feed 352of FIG. 3) from the assistance device 200 and/or the sensing device 300,and/or initiate a two-way audio and/or visual communication channelbetween the device 700B and the assistance device 200 and/or the sensingpod 300. The device 700C may have a similar capability.

In addition to generating the alert message 122, additional instances ofthe emergency action 120 include but are not limited to automaticallyestablishing the audio feed 350 and/or the video feed 352, changingparameters of the safety profile 114 such as under what conditions aresponse of the user 100 will be determined to be an event of concern116 and/or under what conditions the event of concern 116 will bedetermined to be the adverse event 118, and altering which serverprimarily receives and parses the voice communications of the user 100(e.g., the automated assistance server 400 or the emergency server 500).For example, one type of emergency action 120 specified in the emergencyaction data 121 may be to re-route the voice communication 110 of theuser 100 to the emergency server 500 such that an emergency speechrecognition engine 560 is primarily utilized to determine speech and/orspeech patterns of the user 100 (e.g., a stressed tone of voice, lowertolerance for determination that a word sound signal should berecognized as “help”, recognition of profanity or expletives, etc.).

In another example, the emergency action 120 may adjust what futureinstances of the emergency action 120 should be taken upon occurrence ofthe adverse event 118 (e.g., by altering the instructions or data in theemergency action data 121 and/or the safety profile 114). For example, afirst instance of an adverse event 118 may initiate the emergency action120 of sending the alert message 122 to a first caregiver 130A (notshown in the embodiment of FIG. 1). The emergency action 120 in thiscase may also update the safety profile 114 such that the next instanceof the adverse event 118 may expand a scope of a next instance of thealert message 122. For example, when the emergency action 120 istriggered again, both the first caregiver 130A and a second caregiver130B may receive the alert message 122.

In one or more embodiments, the scheduling server 600 schedules andinitiates assistance actions 478 of assistance applications 478, asshown and described in conjunction with FIG. 4 and FIG. 6, and theprocess flows of FIG. 18 and FIG. 19. The scheduled instances of theassistance action 478 may allow for streamlined interaction from aperspective of the user 100, especially where the user 100 is physicallyimpaired, may have difficulties articulating words, and/or may havedifficulties with their memory. However, the scheduling server 600 mayalso provide a synergistic opportunity for interacting with the user 100such that the state of the user 100 can be determined. For example,where the assistance device 200 and/or the emergency server 500 areprogrammed to evaluate appropriate responses of the user 100, a voiceinput to the assistance application 470 (e.g., the voice input 480 ofFIG. 4) may be used to determine the negative response event 115. Forexample, where an assistance application 470 for the delivery of foodmay ask for an address, the assistance device 200 may request the streetaddress from the user 100. If the user 100 responds with an address notmatching the street address associated with the user 100 or a streetaddress where the user 100's device 700A is not currently located, thenthe response of the user 100 may be determined to be a negative responseevent 117. In this case (e.g., where such analysis has been defined inthe instructions of the assistance device 200 and/or the emergencyserver 500), a request for the voice input 480 by the assistance device200 acts as the verbal status query 108 to the user 100.

Although not shown in the embodiment of FIG. 1, there may be multipleinstances of the user 100 which may be individually authenticated by avoice pattern, a device 700A, or other means of identification to ensurethe user 100 is matched to their corresponding instance of the safetyprofile 114. For example, the caregiving environment may have multipleresidents, or an elderly couple may live inside a home together.However, in one or more embodiments there may be a standard version ofthe safety profile 114 that applies to two or more of the users 100within a multi-user caregiving environment. The caregiver 130 may alsoreside in the caregiving environment or generally remain on-site (e.g.,at a nursing home). Similarly, there may be coordinated within thecaregiving environment multiple instances of the assistance device 200and/or the sensing pod 300 to provide an extended environment (e.g., theextended environment 103) through which the one or more users 100 maycommunicate with aspects of the assistance coordination network ofFIG. 1. In one or more embodiments the assistance device 200 and eachinstance of the sensing pod 300 may form a wireless mesh network where,for example, WiFi from one or more wireless routers may have troublecovering some areas of the caregiving environment. In addition there maybe multiple instances of the caregiver 130 or the emergency service 150,each with one or more instances of the device 700. In one or moreembodiments there may additionally be multiple instances of theautomated assistance server 400, either accessible through a commonprovider of voice controlled assistance (e.g., Amazon Alexa) or throughmultiple providers of voice controlled assistance (e.g., Amazon Alexaand Apple Siri).

FIG. 2 illustrates the assistance device 200 according to one or moreembodiments. Each component shown within the sound assistance device 200is connected to each other component of the assistance device 200, forexample through a data bus, wiring, or other data communication systemsand methods. The assistance device 200 is communicatively coupled to thenetwork 101 is surrounded by an immediate environment 102. For examplethe immediate environment may be a single room, a radius of 20 feet fromthe assistance device 200, or a radius of 70 feet from the assistancedevice 200. The assistance device 200 includes a processor 201 that is acomputer processor and a memory 203 that is a physical computer memory,each of which are communicatively coupled to each of the components ofFIG. 2. The assistance device 200 may include a presence sensor 202 thatmay be specifically used to sense a presence of the user 100 to generatedata usable to determine the sensing event 104. For example, thepresence sensor 202 may be a motion sensor, an infrared sensor, an RFIDsensor that senses proximity of a radio frequency identification chip, anoise sensor, a light sensor, a vibration sensor, a Bluetooth sensorthat senses proximity of the device 700 of the user 100 through aBluetooth connection, and/or an ultrasonic frequency sound sensor thatdetects an ultrasonic sound frequency emitted from the device 700 of theuser 100. The assistance device 200 includes a speaker 204 that cangenerate the automated communication 106 and a microphone 206 that canreceive a sound 105 from the immediate environment 102. For example, thesound 105 may include an ambient sound, a voice response of the user100, and/or additional noises. In one or more embodiments the microphone206 may be used as the presence sensor 202 when the sensing algorithm208 is configured to analyze sound of the immediate environment 102 todetermine the presence of the user 100. The assistance device 200 alsoincludes a network interface controller 210 for communication over thenetwork 101. The network interface controller 210 may implement theelectronic circuitry required to communicate using a specific physicallayer and data link layer standard such as Ethernet, Fibre Channel,Wi-Fi or Token Ring and may allow for the connection of the assistancedevice 200 to each of the other devices and servers through the network101 within the assistance coordination network of FIG. 1. The assistancedevice 200 may include a display 109 that may display text and/orgraphics to the user 100, including a graphical user interface. Forexample, the display 109 may be a liquid crystal display (LCD) screen.

The sensing algorithm 208 is a set of computer executable instructionsusable by the processor 201 to determine with a high probability apresence of the user 100 based upon data inputs from the presence sensor202. For example, the sensing algorithm 208 may determine a presence ofthe user 100 when a signal strength of an electromagnetic signal reachesa threshold strength, when “pings” of the device 700 of the user 100within a timeframe reach a threshold number, and/or when the sound 105of the immediate environment 102 raises above a threshold level that maybe calculated statically (e.g., set to a decibel level) or re-factoredcontinuously (e.g., adjusted based on the ambient sound of the immediateenvironment 102). Upon the sensing event 104 that may have a high degreeof certainty in determining a presence of the user 100 within theimmediate environment 102 the sensing algorithm 208 may generate data tobe acted on by a status query procedure 210 or initiate additionalactions such as to log the sensing event 104 and/or add the sensingevent 104 to the log 113.

Determination of the sensing event 104 may initiate a procedure call orotherwise transmit data to the status query procedure 210. The statusquery procedure 210 is a set of computer executable instructions usableby the processor 201 to determine whether to initiate the status query107. The status query 107 may be a question or interrogatory generatedas data and usable to elicit a response to determine a state, adisposition, a feeling, an emotion, a condition (e.g., physical, mental,psychological, and/or emotional) and/or a status of the user 100. Forexample, the status query 107 may be a text message sent to a device 700of the user 100. In one or more embodiments, the status query 107 is theverbal status query 108 generated by the speaker 204 of the assistancedevice 200.

The status query procedure 210 may determine under what circumstancesand what kinds of the status query 107 to generate. For example, wherethe sensing event 104 is determined through detection of the sound 105,the status query procedure 210 may generate the verbal status query 108,whereas if the sensing event 104 is determined through a GPS coordinateof a device 700 of the user 100 (e.g., the GPS coordinate of a mobiledevice 750) the status query 107 may be the text message sent to thedevice 700 of the user 100.

The status query procedure 210 may also include computer executableinstructions that when executed on the processor 201 determines that thestatus query 107 should be expanded from the immediate environment 102of the assistance device 200 (and/or the immediate environment 102 of asensing pod 300 that generated data to indicate the presence of the user100) to the extended environment 103 (e.g., all instances of the sensingpod 300 communicatively coupled to the assistance device 200 in thecaregiving environment may repeat the verbal status query 108). Forexample, following a non-response event 117, the status query procedure210 may repeat the verbal status query 108 within the extendedenvironment 103 and additionally generate the status query 107 as a textmessage send to the mobile device of the user 100.

The countdown timer 212 is a software and/or hardware timer initiatedduring certain actions of the assistance device 200. For example, upongenerating the status query 107 the countdown timer 212 may be set to atime length (e.g., 1 minute, 10 seconds, 1 second, 900 milliseconds) bywhich the user 100 must begin to provide or completely provide the voicecommunication 110 that is a verbal response. In one example, thecountdown timer 212 of four seconds is initiated upon generation of theverbal status query 108. Failure of the user 100 to generate any soundmay then result in a non-response event 117 and/or the extension of theverbal status query 108 to the extended environment 103. In a morecomplex example, the verbal status query 108 of “What time of day isit?” may initiate a 20 second timer in which the user 100 must give theverbal response before determination of a non-response event 117, butalso initiate a 10 second timer the expiration of which results in anegative response event 115 even if the user 100 answers beforeexpiration of 20 seconds (e.g., the user 100 was responsive but wasunable to tell the time quickly, possibly indicating cognitiveconfusion). Additionally in this example, failure to answer the correcttime by plus or minus fifteen minutes could result in a determination ofa negative response event 115. This example may be a form of a quiz, apuzzle, or a trivia as shown and described in conjunction with FIG. 23.

The assistance service coordinator 214 is a set of computer executableinstructions that when executed on the processor 201 cause the processor201 to allocate data communications between the automated assistanceserver 400 and the emergency server 500. In one or more embodiments:under certain predetermined conditions the assistance servicecoordinator 214 may transmit all or almost all instances of the voicecommunication 110 of the user 100 to the automated assistance server 400to provide the general assistance; whereas under other predeterminedconditions the assistance service coordinator 214 may route all orsubstantially all of the voice communication 110 to the emergency server500.

In one or more embodiments, upon the adverse event 118 based on one ormore instances of the non-response event 117, the assistance servicecoordinator 214 may terminate communications with the automatedassistance server 400 and transmit any subsequently received instance ofthe voice communication 110 to the emergency server 500 through thenetwork 101. Similarly, the assistance service coordinator 214 mayre-route communications upon an event of concern 116 and/or in responseto the emergency action 120. Where a two-way communication isestablished between the device 700B of the caretaker 130 and theassistance device 200 the user 100 the assistance service coordinator214 may terminate all communication with the automated assistance server400 such that the interaction between the caregiver 130 and the user 100is not adversely affected by the automated communication 106. In one ormore other embodiments, the assistance service coordinator 214 may alsoforward a voice communication 110 to both the automated assistanceserver 400 and the emergency server 500. Where the emergency speechrecognition engine 560 determines that the forwarded data includes oneor more emergency words, phrases, stressed voice tones, or otherindicators of an adverse event 118 requiring execution of the emergencyaction 120, then the emergency server 500 may communicate a signal tothe assistance service coordinator 214 to re-route all voicecommunication 110 to the emergency server 500. The re-routing may occurby changing an application layer communication protocol for packaging anaudio file or audio data stream containing the voice communication 110of the user 100 and changing a destination address on the network 101(e.g., an internet protocol (IP) address).

The activity detection module 216 is a set of computer executableinstructions that when executed on the processor 201 determine athreshold inactivity level of the user 100 by receiving data from thedevice 700 of the user 100, the sensing pod 300 and/or another source.In one or more embodiments, the activity detection module 216 receivesaccelerometer data from the device 700 of the user 100 (e.g., theaccelerometer 725 of a mobile device 750) to determine the user 100 islikely moving and therefore is likely in an active state. A thresholdinactivity may also be determined based on movement of a GPS coordinateof the device 700 of the user 100, a network traffic of the device 700of the user 100 (e.g., internet browsing activity on a desktop computerwithin a 24-hour period), the microphone 706 of the device 700 of theuser, and/or graphical analysis or light intensity analysis on a videofeed and/or video camera 720 of the device 700 of the user 100.

The speech-text converter 218 is a set of computer executableinstructions that when executed on the processor 201 converts data in atext format (e.g., a text string, a text file) into an audio format,and/or may convert the voice communication 110 and/or the automatedcommunication 106 into a text format via a limited speech recognitioncapability and/or basic sound matching with stored audio files. Forexample, the speech-text converter may be used to convert a voicecommunication by the emergency service 150 into a text format fordisplay to the user 100 on the display 209 of the device 200.Conversely, the caregiver 130 may send a text message from the mobiledevice of the caregiver 130 to the assistance device 200 that convertsinto a speech communication to the user 100 via the speaker 204.

The assistance device 200 may also include a video camera 200 forrecording a video signal for processing, analysis, storage and/ortransmission by the assistance device 200. In one or more embodimentsthe video camera 220 begins to generate a video feed 352 stored and/orcommunicated to the device 700B of the caregiver 130 and/or the device700C of the emergency service 150 upon determination of the adverseevent 118.

The assistance device 200 may also include a number of other sensors toimprove monitoring and safety of the user 100, including detectionwithin the caregiving environment of the user 100 a health risk to theuser 100. In the embodiment of FIG. 2, the assistance device 200includes a particulate sensor 222 and a chemical sensor 224. Theparticulate sensor 222 may detect a particulate 223 within the immediateenvironment 102. For example the particulate 223 may include smoke,steam, smog, dust, soot and/or other harmful suspensions in air. In oneor more embodiments the particulate sensor 222 is a smoke detector andthe particulate 223 is smoke. The chemical sensor 224 may detect achemical 225 that me be harmful to the user 100 at one or moreconcentrations. For example, the chemical 225 may be carbon dioxide,carbon monoxide, cyanide gas, ammonia, oxygen (including lack of or overabundance of oxygen), ozone, nitrous oxides, sulfurous oxides, volatileorganic compounds, and similar hazardous molecules. Although not shownin the embodiment of FIG. 2, a biological sensor (e.g., to detect amold, a bacteria, a fungi) may also be employed to detect environmentalhazards.

In addition to hardware elements and computer executable instructionsthe assistance device 200 includes numerous data within a database 205that may be stored on the memory 203 (e.g., random access memory (RAM),a storage device (e.g., a disk, a solid-state memory), a volatilememory, a memrister, a solid-state memory). The database 205 may storeone or more instances of the status query 107, for example a basicphrase that may be read and initiated as the verbal status query 108(“How are you?”) by the status query procedure 210. The status query 107may also be transmitted through the network 101 from the automatedassistance server 400, the emergency server 500 and/or the schedulingserver 600 in which case the status query 107 may exist temporarily orinstantaneously within the database 205 and/or the memory 203. Thestatus query 107 may be stored as an audio file, or may be stored as atext file that may be converted by the speech-text converter 218 intothe verbal status query 108.

The database 205 may further include the safety profile 114 of the user100. The safety profile 114 comprises conditions under which one or moreevents of concern 116 constitute an adverse event 118, and may alsoinclude the emergency action data 121 to be taken upon determination ofa particular type of the adverse event 118 defined and stored as theemergency action data 121. Although shown separate in the embodiment ofFIG. 2, the emergency action data 121 may be associated with and/orstored within the safety profile 114. In one or more embodiments, thesafety profile 114 may specify a decision tree and/or a complex decisiontree for determination of which instances of the emergency action 120are to be taken in response to an adverse event 118 (e.g., whichinstances of the emergency action data 121 should be executed underwhich circumstances). For example, one process by which determination ofinstances of the emergency action 120 are taken is shown and describedin conjunction with FIG. 15. The safety profile 114 may be unique to aparticular instance of the user 100 and custom built for the particularuser 100's needs and/or concerns. The safety profile 114 may also beselectable and updatable by the caretaker 130 based on several availablepre-generated profiles. Additionally, the safety profile 114 may bestandard for all or many instances of the user 100 (e.g., a standardfactory setting that may have shown a good result or a result with a lownumber of false events of concern 116). In addition, the safety profile114 may be continuously updated through updates from the network 101and/or through machine learning algorithms that may adjust to patternsor activities of an instance of the user 100 (e.g., a non-response event117 from one particular instance of the sensing pod 300 within thecaregiving environment of the user 100 is not considered an event ofconcern 116 when a high level of ambient noise exists within theimmediate environment 102, for example because a television may createtoo much noise for the user 100 to hear the verbal status query 108).

The log 113 is a list of responses of the user 100 and may includemetadata associated with the responses such as a time of the response, arisk score of the response, an assistance device 200 and/or a sensingpod 300 from which the response was received, etc. The log 113 mayinclude responses of the user 100 including various queries translatedinto text or stored as audio files and/or designations of the negativeresponse event 115, the non-response event 117, and/or the positiveresponse event 119. The log 113 and/or parts of the log 113 may becommunicated to the device 700 of the caretaker 130, the device 700 ofthe emergency service 150, and/or even the device 700 of the user 100.The log 113 may also be backed up or even primarily stored on theemergency server 500. The log 113 may also store the adverse event 118and associated meta data (or the adverse event 118 may be stored in aseparate log).

The database 205 may additionally store an alert message 122 generatedas one possible instance of the emergency action 120. The alert message122 may include the log 113 and/or data of the log 113 such as one ormore events of concern 116 giving rise to the determination of theadverse event 118. For example, the alert message 122 may be themessage: “Hello. Janet has been non-responsive during two status checkstoday. The first check-in was at 9:02 AM. The second check-in was at11:46 AM”.

In one or more embodiments, the scheduling server 600 as shown anddescribed in conjunction with FIG. 6 may store in the database 605 aninvocation command 472 and an initiation command 476 of an assistanceapplication 470. However, the invocation command 472, the initiationcommand 476, and/or one or more voice inputs 480 may also be stored inthe database 205. Where stored in the database 205, the schedulingserver 600 may transmit the action instruction 232 to the assistancedevice 200 to initiate the invocation command 472, the initiationcommand 476, and/or any voice inputs 480, as transmitted from theassistance device 200. In other words, a condition triggering theassistance application 470 may occur on the scheduling server 600 asshown and described in conjunction with FIG. 6, but the assistanceapplication 470 may be invoked from the assistance device 200.

The database 205 may include a textual message 228 that may be a textversion of the automated communication 106 to be presented to the user100 on the display 209 of the assistance device 200. For example, thetextual message 228 may be generated by the automated assistance server400 or converted from an audio speech communication of the automatedassistance server 400 with the speech-text converter 218. Similarly, theemergency server 500 may transmit the textual message 228 and/or dataconverted into the textual message 228 for display to the user 100 in agraphical user interface. The textual message 228 can also be forwardedto the device 700 of the user 100.

The embodiment of FIG. 2 further may include one or more pieces of datafor automatically invoking and/or streamlining activation of one or moreinstances of the assistance application 470, including an initiationcondition 230 of an assistance action 478, an action instruction 232, acustom command word 234, and one or more voice inputs 236 requiredbefore completion of the execution of the assistance action 478. Thisdata is described in conjunction with the assistance server 400 of FIG.4. Additionally, the assistance device 200 may further include an awaymessage 244 of the user 100 that is stored as an audio recording 242 andan expected inactivity time 240 that is a length of time specified asdata associated with the audio recording 242. The expected inactivitytime 240 may be usable to suspend instances of the status query 107,usable as notification to the caregiver 130 of an activity of the user100 away from the caregiving environment, and/or for additionalpurposes. The away message 244 and the expected inactivity time 240 areshown and described in conjunction with the process flow of FIG. 22.

FIG. 3 illustrates a sensing pod 300 that can be communicatively coupledto the assistance device 200 of FIG. 2 to enable an extended environment103 for which the assistance device 200 can perceive and/or communicatewith the user 100, the sensing pod 300 having its own immediateenvironment 102 (e.g., the immediate environment 102B of FIG. 1) of thesensing pod 300, according to one or more embodiments. The immediateenvironment 102B may or may not overlap with the immediate environment102A. Each component shown within the sensing pod 300 is connected toeach other component of the sensing pod 300, for example through a databus, wiring, or other data communication systems and methods.

In one or more embodiments, the sensing pod 300 may act as a less costlyapparatus to extend a capability of the assistance device 200 to coverall or the most important and/or frequented areas of the caregivingenvironment. For example, within a caretaking environment there may beone instance of the assistance device 300 and a number of instances ofthe sensing pod 300 (e.g., one, five, twenty) communicatively coupledwith the assistance device 200 through the network 101. In one or moreembodiments, each sensing pod 300 and the assistance device 200 may havetheir network interface controller 207 and network interface controller307, respectively, capable of interfacing with a wireless router (e.g.,WiFi) that may form a local area network, with an additional connectionbetween the assistance device 200 and a wide area network providingaccess to the rest of the assistance coordination network of FIG. 1. Inone or more embodiments, the sensing pod 300 may powered by a battery.

As shown in FIG. 3 the sensing pod 300 includes a processor 301 and amemory 303. The sensing pod 300 may include the speaker 304 to generateautomated communication 106 that may be received through the network 101and/or the microphone 306 that may receive the sound 105 from theimmediate environment 102 of the sensing pod 300 and either process asignal of the sound 105 at the sensing pod 300 or transmit the signal ofthe sound 105 to the assistance device 200. The sensing pod 300 mayfurther include hardware components similar to the assistance device200, including the presence sensor 302, the display 309, the videocamera 320, the particulate sensor 322, and/or the chemical sensor 324.The network interface controller 307 may be a wirelesstransmitter-receiver, and may be able to form a mesh network between andamong the assistance device 200 and each of the other instances of thesensing pod 300 within the caregiving environment.

The sensing algorithm 308 may function similarly to the sensingalgorithm 208 or may be modified to better perform its function as partof the extended network 103 of the assistance device 200. For example,upon receive a signal from the presence senor 302 the sensing algorithm308 may determine if the signal constitutes the sensing event 104, orthe sensing algorithm 308 may merely determine whether the signal mightqualify as a sensing event 104 and relay the signal to the assistancedevice 200 for determination by the sensing algorithm 208. As shown inthe memory 303, the sensing pod 300 may buffer the audio feed 350, thevideo feed 352, and any automated communication 106 and/or voicecommunication 110.

FIG. 4 illustrates the automated assistance server 400 of FIG. 1 thatmay provide a general assistance to the user 100, according to one ormore embodiments. For example, the general assistance may be a personalassistance for initiating a web search, ordering a product, engaging ina social interaction (either with an artificial intelligence assistantimplemented in software and/or establishing a peer-to-peer connectionwith other instances of the user 100), schedule a reminder or event,control household connected devices and/or appliances (e.g., “internetof things” or “IoT” devices usable by the user 100 within a household),and similar functions. The automated assistance server 200 includes aspeech recognition engine 460 to recognize and/or translate a speech ofthe user 100 and a personal assistance engine 462 that is a computerprogram for providing the personal assistance, carrying out actions atthe request of the user 100, and for generating speech responses for theuser 100. The speech recognition engine 460 may be software capable ofidentifying the sounds produced in human speech and/or capable ofreceiving and interpreting dictation. The speech recognition engine 460may forward results of analysis to the personal assistance engine 462 toproduce an appropriate response and/or induce an appropriate actionaccording to a voice command protocol of the personal assistance engine462. In one or more embodiments, the personal assistance engine 462 mayprovide an artificial intelligence assistant (e.g., Siri, Alexa, GoogleAssistant, Cortana, Bixby) and/or a robotic personality for interactionwith the user 100. For example, the personal assistance engine 462 mayfeature a natural language user interface to answer questions, makerecommendations, and perform actions by delegating requests to a set ofweb services. The personal assistance engine 462 may adapt to the user100's individual language usage and individual searches (preferences)with continuing use, and may returns results that are individualized. Inone or more embodiments, the personal assistance engine 462 may also becapable of music playback, making to-do lists, setting alarms, streamingpodcasts, playing audiobooks, and providing weather, traffic, and otherreal time information, such as news. The personal assistance engine 462may also be able to control several smart devices as the “brains” of ahome automation system. In one or more embodiments not shown in thefigures, the assistance device 200 may include a software componentand/or client-side application for efficient communication andinterfacing with the automated assistance server 400. Each componentshown within the automated assistance server 400 is connected to eachother component of the automated assistance server 400, for examplethrough a data bus, wiring, or other data communication systems andmethods.

The automated assistance server 400 includes a processor 401 on whichcomputer executable instructions are processed and a memory 403. Inaddition, the automated assistance server 400 in one or more embodimentsmay include one or more instances of an assistance application 470 thatis computer software that can perform specific instances of theassistance action 478 in response to a voice command of the user 100(e.g., the initiation command 476) conforming to a voice commandprotocol 474 of the assistance application 470. In one or moreembodiments, the assistance application 470 may be referred to as a“skill” of the automated assistance service (e.g., an Amazon AlexaSkill) that the user 100 may have to install or enable for use with theautomated assistance service.

For example, the assistance application 470 may be a history applicationthat may be able to tell the user 100 about historical events. Thehistory application may have an invocation command 472, for example“launch history app.” The voice command protocol 474 may be one or morecommands that the history app recognizes and will response to. In thiscase, the voice command protocol 474 may include a number of commandssuch as “Tell me about another event from September 3rd” or “Hear moreabout this event” or “tell me about a famous battle in 1840”. Eachcommand acts as initiation command 476 conforming to the voice commandprotocol 474. The assistance application 470 may additionally requireone or more voice inputs 480. For example, if the user 100 asks for “afamous battle” the assistance application 470 may request a voice input480 of “in which year would you like to hear about a famous battle?”Similarly, for an assistance application 470 for ordering food delivery,an initiation command 476 may always request voice inputs 480 thatinclude the menu items to be ordered and/or an address for delivery.Streamlined interactions involving the assistance application 470between the assistance device 200, the automated assistance server 400,and the scheduling server 600 are further shown and described inconjunction with FIG. 6, FIG. 18 and FIG. 20.

FIG. 5 illustrates the emergency server 500 of FIG. 1 providing aspecialized emergency service to the user 100 and/or capable of carryingout some of the functions shown and described in conjunction with theassistance application of FIG. 2, according to one or more embodiments.The emergency server 500 includes a processor 501, a memory 503, thedatabase 505, and is connected to the network 101. Each component shownwithin the automated emergency server 500 is connected to each othercomponent of the emergency server 500, for example through a data bus,wiring, or other data communication systems and methods.

The emergency speech recognition engine 560 may be similar to the speechrecognition engine 460 but with enhanced and/or emphasized capability torecognize, interpret and/or identify an emergency speech 561 of the user100. For example the emergency speech 561 may be: an emergency word suchas “help!” (e.g., the emergency word 1110 of FIG. 11); a speech tonetending to indicate anxiety, pain, or suffering; and/or abnormallyrapid, loud or incoherent speech. The emergency speech recognitionengine 560 may also have a lower threshold for recognizing certain wordscommonly associated with danger or injury to the user 100, for example“medication”, “dizzy”, “injury”, “smoke”, explitives, and similar words.In one or more embodiments, the emergency speech recognition engine 560may simultaneously receive the voice communication 110 of the user 100at the same time as the speech recognition engine 460; however, uponmeeting a threshold number of words associated with an emergency, dangeror injury to the user 100 the automated emergency assistance engine 562may communicate with the assistance coordinator 214 to routecommunications primarily to the emergency server 500 and/or initiatefrequent instances of the status query 107 to the user 100.

The automated emergency assistance engine 562 is a software program thatmay provide a specialized emergency assistance to the user 100 and mayprovide an automated emergency assistance service. For example, theautomated emergency assistance engine 562 may include procedures foraffirmatively asking the user 100 if they are alright or if the user 100needs assistance. Upon satisfaction of threshold conditions (forexample, the automated emergency assistance engine 562 generates anautomated communication 106 asking the user 100 if there is a fire andthe user 100 responds “yes”), the automated emergency assistance engine562 may first contact the emergency service 150 (e.g., call 9-1-1) andmay additionally provide succinct instructions to the user 100 based onknown factors to reduce risk to the user 100 under the circumstanceuntil a live person associated with the emergency service (e.g., a 9-1-1operator, a poison control specialist, a physician) may be connectedthrough a two-way communication between the assistance device 200 andthe device 700 of the emergency service 150.

The automated emergency assistance engine 562 may also take anyadditional emergency action 120, similar to those shown and described inconjunction with FIG. 1 and FIG. 2. For example, the automated emergencyassistance engine 562 may generate the alert message 122 to transmit tothe device 700B of the caregiver 130 and/or the device 700C of theemergency service 150. As shown in the embodiment of FIG. 5, thedatabase 505 may store one or more instances of the safety profile 114,the log 113 (including without limitation the negative response event115, the non-response event 116 and the positive response event 119),the event of concern 116, the adverse event 118, the alert message 122and/or the emergency action 120. In one or more embodiments theassistance device 122 may transmit the voice communication 110 of theuser 100 to the emergency server 500 for determination of the event ofconcern 116 and/or the adverse event 118 requiring the emergency action120. In one or more other embodiments, the assistance device 200 maygenerate the adverse event 120 that may be communicated to the emergencyserver 500, which may in turn generate the alert message 122 and/or takeadditional instances of the emergency action 120. In yet one or moreother embodiments, the assistance device 200 may determine one or moreevents of concern 116 and transmit the events of concern 116 to theemergency server 500 for determination of the adverse event 118. Theemergency server 500 may also store as data what is determined by theemergency speech recognition engine 560 to be the emergency speech 561of the user 100, and/or the emergency speech data 563 that may be acatalogue of reference emergency speech usable by the emergency speechrecognition engine 560. The emergency speech data 563 may be unique tothe user 100 or common to many instances of the user 100. If unique, theemergency speech data 563 may be stored in the safety profile 114 of theuser 100.

FIG. 6 illustrates the scheduling server 600 of FIG. 1 including aprocessor 601, a memory 603, a database 605, and an assistance servicescheduler 690 for streamlining, scheduling and/or enhanced communicationand/or interaction with the automated assistance service provided by theautomated assistance server 400. In one or more embodiments, a functionof the scheduling server 600 is to pre-generate interactions withinstances of the assistance application 470. For example, one routinemay determine that at 6 PM on each weekday the user 100 will be askedvia the assistance device 200 (and/or the sensing pod 300) whether theuser 100 would like to order dinner. Upon an affirmative response by theuser 100 the assistance service scheduler 690 may look up an assistanceapp identifier 692 with an associated invocation command 472 and look upthe assistance action identifier 692 associated with an initiationcommand 472. In the restaurant application example above, the assistanceapp identifier 692 may be data that is a unique value specifying a pizzarestaurant (e.g., Domino's Pizza), the invocation command 472 may be“order from Domino's,” the assistance action identifier 692 may be aunique identifier associated with a particular command within the voicecommand protocol 474 of the pizza app, and the initiation command 472may be “order pizza.” While ordinarily the user 100 would be asked foradditional voice inputs 480 in queries such as “please select a size” or“please select toppings” or “please provide delivery address”, the voiceinputs 480 may be pre-stored in the database 605 and communicated to theassistance application 470 following the initiation command 476. In thisexample, the user 100 did not have to remember to initiate theinteraction to order dinner, did not have to remember the invocationcommand 472, did not have to remember the initiation command 476, anddid not have to manually enter any repetitive or redundant instances ofthe voice inputs 480. From the experience of the user 100, they wereasked if they would like dinner; the user 100 responded affirmatively;and the user 100 was then delivered a pizza to an address of thecaregiving environment.

In one or more embodiments a custom command word 234 may be associatedwith the routine. For example the user 100 may specify “order me pizza”to mean run the entire routine for ordering a 10-inch Dominos peperonipizza and delivering it to the address of the caregiving environment.The custom command word 234 may be stored as an audio file or a textfile on the assistance device 200 and associated with a uniqueidentifier matching a record on the scheduling server 600, the recordcomprising the assistance app identifier 692, the assistance actionidentifier 694, the invocation command 472, the initiation command 476and/or one or more pre-stored voice inputs 480. The custom command word234 may also be stored on the scheduling server 600 and the voicecommunication 110 of the user 100 relayed to the scheduling server 100for determination of whether the voice communication 110 of the user 100contains the custom command word 234.

In one or more embodiments, one or more of the voice inputs 480 canremain unspecified or be designated for a voice input from the user 100at the time the assistance action is initiated. For example, where theuser 100 moves between two caregiving environments a street addressinstance of the voice input 480 may remain unspecified and asked of theuser 100 immediately following the initiation command 476 of theassistance application 470. The voice input 480 provided by the user 100for the street address may also be utilized as a basis for an event ofconcern 116 or otherwise logged in the log 113. Each component shownwithin the scheduling server 600 is connected to each other component ofthe scheduling server 600, for example through a data bus, wiring, orother data communication systems and methods.

While the assistance device 200 may include each of the data shown anddescribed in the database of 605 and carry out similar functions, in oneor more embodiments the streamlined routines are stored on thescheduling server 600 and an initiation command 230 is utilized by theassistance device 200 to trigger the routine. FIG. 7 illustrates thedevice of FIG. 1 of the user 100, the caregiver 130, and/or theemergency service 150. The device 700 may be, for example, a mobiledevice (e.g., an iPhone, an Android phone, a Google phone, a Windowsphone), a tablet (e.g., an iPad, a Galaxy, Amazon Fire), a wearabledevice (e.g., a smartwatch, an iWatch, a wristband tracking a vitalsign, a necklace with a panic button), a connected household device,and/or a desktop computer. Each component shown within the device 700 isconnected to each other component of the device 700, for example througha data bus, wiring, or other data communication systems and methods. Thedevice of FIG. 7 includes the processor 701, the memory 703, the speaker704, the microphone 706, the accelerometer 725, the GPS unit 726 and/orthe presence signaler 727. The presence signaler 727 may generate asignal detectable by the presence sensor 202 of FIG. 2 and/or thepresence sensor 302 of FIG. 3, according to one or more embodiments. Forexample the presence signaler 727 may be an RFID chip, a Bluetoothcommunications signal generator, or may generate an ultrasonic sounddetectable by the assistance device 200 and/or the sensing pod 300. Thevideo camera 720, the speaker 704, and/or the microphone 706 may beusable to establish communication between and among the user 100, thecaregiver 130, and/or the emergency service 150 as one of the one ormore emergency actions 120 coordinated by the assistance device 200.Data of the accelerometer 725 and/or the GPS unit 726 may be analyzed bythe activity detection module 216. In one or more other embodiments, thedevice 700 does not include all components shown in FIG. 7: a desktopcomputer generally may not include the accelerometer 725.

FIG. 8 illustrates aspects of the assistance coordination network ofFIG. 1 deployed inside and outside of a caregiving environment 800utilizing a local area network (LAN 801A) to communicatively couple theassistance device 200A, the sensing pod 300B and the sensing pod 300C,according to one or more embodiments. For example the caregivingenvironment 800 may be a home of the user 100, with the assistancedevice 200A placed in a kitchen, the sensing pod 300B placed in ahallway between a bedroom and a bathroom, and the sensing pod 300Cplaced in a living room, each connected to the other through the LAN 801that may be a wireless router (e.g., connected using an 802.11 wirelesscommunication protocol). The wireless router implementing the LAN 801Amay then be connected to the WAN 801 (e.g., the internet) via anEthernet cable and cable modem accessing an internet service provider(ISP). The assistance device 200A may have the immediate environment102A (e.g., the kitchen), the sensing pod 300B may have the immediateenvironment 102B (e.g., the bedroom, hallway, and bathroom), and thesensing pod 300C may have the immediate environment 102C (e.g., theliving room). Together, the immediate environment 102A, 102B and 102Cmay form the extended environment 103 of the assistance device 200A.

In the embodiment of FIG. 8, the user 100 and/or a device 700 of theuser 100 (e.g., the mobile device 750A, the wearable device 752) may besensed in the sensing event 104 upon the user 100 nearing or enteringthe bathroom of the caregiving environment 800 and additional actionsmay be taken as shown and described in conjunction with FIG. 9.Periodically or upon occurrence of a specific event, the log 113 may becommunicated from the assistance device 200A, through the LAN 801A andthe WAN 801B to the device 700 of the caregiver 130 (e.g., the mobiledevice 750B). As shown in FIG. 8 the mobile device 750A may have anadditional method of access to the WAN 801B, for example through a celltower connection. In one or more alternative embodiments each of thecomponents may be connected using a cellular data connection, forexample 4G or LTE, or may even be connected through a satellite internetconnection.

FIG. 9 illustrates a detailed view in which the interaction of FIG. 8may occur in which the sensing pod 300B determines a presence of theuser 100 within the immediate environment 120B through the sensing event104. The sensing event 104 may be determined to be, for example, a noisemade by the user 100 above a threshold environmental sound as gatheredby the microphone 306, and/or by detecting a proximity of the mobiledevice 750A and/or the wearable device 752. In the embodiment of FIG. 9,the sensing pod 300B determines the sensing event 104 and calls theassistance device 200 which may return the voice status query 108 to theuser 100 in the form of the automated communication 106. For example,the user 100 may be asked “Isn't it a nice day?, What do think?”. Theuser 100 may respond with a negative response 125 (e.g., “No”, or “Idon't care”) that may be determined to be the negative response event115 and/or a positive response 129 (e.g., “Yes, it's nice out”) that maybe determine to be the positive response event 119. The negativeresponse event 115 is an event of concern 116 that may logged in the log113 and which may be immediately communicated to the caregiver 130 asshown and described in conjunction with FIG. 8.

FIG. 10 illustrates a second caregiving environment network (e.g., thecaregiving environment 1000) comprising the assistance device 200A andthe sensing pod 300B and demonstrating an instance of the emergencyaction 120 in response to the non-response event 117. Upon generation ofthe verbal status query 100 the assistance device 200A may initiate thecountdown timer 212. In the embodiment of FIG. 10 the non-response 127initially results in expansion of the status query 107 (e.g., a textmessage, a push notification) to the mobile device 750A. Additionally, avolume of the speaker 204 of the assistance device 200A and/or thespeaker 304 of the sensing pod 300B may be raised and the verbal statusquery 108 repeated. Upon continuation of the non-response 127 theassistance device 200A may determine and log an event of concern 116that is the non-response event 117. The assistance device 200A mayreference the safety profile 114 of the user 100 to determine that asingle non-response event 117 is an adverse event 118 requiringexecution of two instances of the emergency action 120. First, the alertmessage 122 comprising the log 113 may be generated and communicated tothe mobile device 750 of the caregiver 130. Second, a sound capture ofthe sound 105 and a video capture of the video 1005 may be initiatedsuch that an the audio feed 350 and the video feed 352 may beautomatically established to stream audio-visual footage from thespeaker 304 of the sensing pod 300B, the video camera 320 of the sensingpod 300, the microphone 206 of the assistance device 200A, and/or thevideo camera 220 of the assistance device 200A. The caregiver 130 mayalso be able to engage in a two-way communication through the speakersof the sensing pod 300B and/or the assistance device 200A to initiatereal-time voice communication with the user 100.

FIG. 11 illustrates a routing of a voice communication 110 of the user100 between the automated assistance server 400 of FIG. 4 providing thegeneral assistance and the emergency server 500 providing of FIG. 5providing the specialized emergency service. At a first time, labeled‘circle 1’ in FIG. 11, the user 100 engages in ordinary, non-emergencyvoice commands and speech with the assistance device 200. For example,the user 100 may ask what the weather will be like tomorrow, or ask whattime a particular sports game will be on television. The automatedassistance server 400 running the personal assistance engine 462 mayprovide automated communication 106 responsive to the questions of theuser 100. During this first time most speech may be communicated to theautomated assistance server 400 but only some speech may me transmittedto the emergency server 500 (e.g., for occasional analysis to determinean emergency speech pattern). However, during a second time designated‘circle 2’, the assistance service coordinator 214 of FIG. 2 maydetermine that communications (e.g., the voice communication 110) are tobe re-routed primarily to the emergency server 500, for example toevaluate the voice communication 110 based on the emergency speechrecognition engine 560. As shown in FIG. 11, the re-routing may occurbecause of a negative response 125 of the user 100, a non-response 127and/or recitation of an emergency word 1110. As an example, the user 100may have fallen during a shower and despite the sensing pod 300 beingnear the bathroom the user 100 may be outside of an ordinary audiblerange of the sensing pod 300. However, after a non-response event 117,more subtle sounds may activate voice analysis and communications may bere-routed to the emergency server 500 which may utilize the emergencyspeech recognition engine 560 that may have an increased sensitivity todetect the emergency word 1110 of the user 100 stuck in the shower.

FIG. 12 illustrates numerous capabilities of the scheduling server 600of FIG. 6 to ease interaction with the automated assistance service andprovide an opportunity for determining events of concern 116. Theassistance action 478 can be initiated upon occurrence of an initiationcondition 230. In one or more embodiments, the initiation condition 230may be the user 100 speaking a custom command word 234. For example, thecustom command word 234 when recited at the assistance device 200 mayautomatically trigger the scheduling server 600 to initiate theassistance action 478 according to a streamlined, pre-defined routine.The assistance action 478 may also be triggered by the sensing event104. For example, the initiation condition 230 may include multipleaspects that must occur simultaneously (e.g., the user 100 sensed in theliving room and the sensing event 104 occurring between 4 PM and 7 PM ona week night).

The scheduling server 600 and/or the automated assistance server 400 maygenerate one or more input requests 1200 for additional voice inputs 480required before completing execution of the assistance action 478. Theuser 100 may respond to the input request 1200 with the verbal input1202. Finally, data in the scheduling server 600 and/or the assistancedevice 200 may indicate that before execution of the assistance action478 that the user 100 is provided with a confirmation request 1204 towhich the user 100 must provide the verbal confirmation 1206 and/or theuser 100 is provided with the authorization request 1208 to which theuser 100 may reply with the authorization 1210 to satisfy theauthorization request 1208. Either the confirmation request 1204 and/orthe authorization request 1208 may be provided to the user 100 beforethe invocation command 472 and/or the initiation command 476 has beentransmitted to the automated assistance server 400. In addition, theuser 100 and/or the caregiver 130 may utilize a web interface or amobile app to adjust and set routines and associated instances of theinitiation condition 230.

FIG. 13 is a monitoring and assistance process flow that illustrates onebasic function of the assistance coordination network of FIG. 1,according to one or more preferred embodiments. Operation 1300 triggersa status query 107 usable to elicit a response from the user 100 foranalysis, the response likely usable to determine a state, adisposition, a feeling, an emotion, a condition (e.g., physical, mental,psychological, and/or emotional) or a status of the user 100. Forexample, the status query 107 may be: a textual message (e.g., thetextual message 228) that can be presented on a display (e.g., thedisplay 209, the display 309, the display 709 as a text message or pushnotification) and/or a graphical user interface on the display; a symbolor an emoji presented on the display (e.g., a smiley face or sad facethat a user 100 may press on a touch screen display); and/or the verbalstatus query 108 generated by a speaker of the assistance device 200,the sensing pod 300, and/or the device 700 of the user 100 (e.g., thespeaker 204, the speaker 304, and/or the speaker 704). Operation 1302receives a response to the status query 107 in a form that may be thesame or different than the format in which status query 107 wasgenerated and/or delivered to the user 100. In one or more embodiments,the format may be the same in both the transmission and the receipt. Forexample, for a status query that is in the form of a textual message 228to the mobile device 750 of the user 100, the user 100 may response witha text message (e.g., the user 100 types “yes fine” on a keyboard of themobile device 750). In one or more other embodiments, the status query107 and the respond to the status query may be in a different format.For example, in response to the verbal status query 108 the user 100 maybe presented with faces having various expressions on the display 209.In such case the user 100 may only need to touch the appropriate facialexpression to generate the response that may be determined to be anevent of concern 116. However, in one or more other embodiments, the“response” received may be the non-response 127, for example an ambientsound of the immediate environment 102 or no touch input.

Operation 1304 parses the response to the status query 107 received inoperation 1302. Operation 1304 prepares data that may be used todetermine the event of concern 116. For example operation 1304 mayreceive and format a sound signal (e.g., from the sound 105 and/or thevoice communication 110), may utilize the speech-text converter 218 togenerate a text string or other data representing the voicecommunication 110, or other formatting processes required to turn rawaudio into data usable by an algorithm.

Operation 1306 determines whether the response of the user 100 is anegative response 125 (e.g., resulting in a determination of thenegative response event 115), a non-response 127 (e.g., resulting in adetermination of the positive response event 117), and a positiveresponse 129 (e.g., resulting in a determination of the negativeresponse event 119). Operation 1308 determines an adverse event 118 hasoccurred based on the event of concern 116 and/or one or more previousinstances of an event of concern 116, the adverse event 118 triggeringone or more instances of the emergency action 120. Operation 1308 mayreference a rule for determining when the event of concern 116 arises tothe adverse event 118. In one or more embodiments, the rule may bestatic (e.g., a set rule for an adverse event 118) that any event ofconcern 116 is an adverse event 118 and the rule may be stored locallyon a device that carried out process 1302. In one or more otherembodiments, a safety profile 114 may be referenced, the safety profile114 including rules and/or conditions under which the event of concern116 is an adverse event 118. For example, the safety profile 114 mayspecify that any negative response event 115 resulting from datagathered from a particular instance of the sensing pod 300 (e.g., thesensing pod located near a stairway where the user 100 may trip) may bedetermined to be an adverse event 118. Operation 1308 may also determinethe emergency action 120 to be taken in association with the adverseevent 118, such as sending an alert message 122 to the device 700 of thecaregiver 130 and/or establishing a two-way communication between thedevice 700 of the emergency server 500. Operation 1308 may determine theemergency action 120 to be taken by referencing the safety profile 114to determine an applicable instance of the emergency action data 121,the referencing and/or extracting the instance of the emergency action121 and reading its contents to determine how to specifically executethe emergency action 120.

Operation 1310 executes an emergency action 120. For example, operation1310 may follow instructions specified in the emergency action data 121to extract data from a log 113 to generate a second instance of the log113 for communication to the device 700 of the caregiver 130, the secondinstance including a subset of data determined relevant to the adverseevent 118. In one instance, a third consecutive instance of thenon-response event 117 may trigger the emergency action 120 of sendingthe alert message 122, but all three instances of the non-response event117 may be communicated in the log 113 within the alert message 122 sentfor communication to the device 700B of the user 100 in the alertmessage 122.

FIG. 14 is an adverse event determination process flow illustrating moredetailed aspects of the process flow of FIG. 13, according to one ormore embodiments. Operation 1400 demonstrates a specific initiation ofthe status query 107 based on sensing the presence of the user 100.Specifically, the presence sensor 202 and/or the presence sensor 302 maysense the user 100 with an immediate environment 102 of the presencesensor 202 and/or an immediate environment of the presence sensor 302.Similarly, the device 700 may sense the presence of the user 100 withinan immediate environment 102 of the device 700 through one or moresensors of the device 700 such as the accelerometer 725 (in the case ofthe accelerometer 725, the device 700 may primarily detect actualphysical contact of the user 100 with the device 700). In one or morealternate embodiments, a different initiator of the status query 107 maybe used, for example if a sensor detects a sound spike and a vibrationspike within the immediate environment 102 of the sensor which mayindicate that the user 100 has fallen down or collided with an object.Operation 1402 generates a status query (e.g., the status query 107, thevoice status query 108) that may be an audible and/or a visualcommunication intended for the user 100. Operation 1404 may initiate acountdown timer 212 set to a time length. There may be a value ‘X’ setto a number of attempts to communicate the status query 107 to the user100. Where no response is received within the time length, operation1406 may increment a data counter and attempt to expand the status query107 into one or more additional devices that can communicate and/orreceive communications from the user 100 (e.g., the assistance device200, the sensing pod 300, the device 700 of the user). Operation 1402may then re-generate the status query 107 in the expanded environment(which may be, for example, the extended environment 103 of FIG. 1). Ifthe counter exceeds the value of attempts (e.g., the ‘X’ value in FIG.14), then operation may proceed to operation 1414. However, whereoperation 1404 receives a signal (e.g., a text message reply, a voiceresponse of the user 100, the user 100 pressing a button) then receiveddata of the response is passed to operation 1408 which parses theresponse.

In one or more embodiments operation 1408 parses the response locally onan apparatus that generated the status query 117 (e.g., the assistancedevice 200) or the response may be parsed in a remote locationaccessible over the network 101, for example the emergency server 500(e.g., which may be a cloud computing server). In some instances, thedata may be in a readily usable format, for example a text string fromthe mobile device 750 of the user 100. In other cases the response mayrequire additional analysis. For example, the voice communication 110may require analysis by the speech recognition engine 460 and/or theemergency speech recognition engine 560. In one or more operations notshown in the embodiment of FIG. 14, any parsed data determined notlikely to be from the user 100 may return the process flow to operation1406.

Operation 1410 analyzes the response to see if it is a negative response(e.g., the negative response 125) which may result in a determinationand storage of a negative response event 115. For example, the verbalresponse of the user 100 may be matched to known words associated with anegative state, for example “No,” “Bad”, “Poor”; a small combination ofwords such as “Not well”; and/or a phrase “I have been better.” In oneor more embodiments, each verbal status query 108 may have associatedspecific known words of negativity (e.g., when asking how a specificailment or injury of the user 100 is doing such as a knee injury theverbal status query 108 may have associated a data that will determine“inflexible” to be a negative response event 115). In one or more otherembodiments the tone of the user 100, the cadence of the user 100, oradditional voice analysis techniques may be used to determine that evenwhere an event may include positive content (e.g., “yes, I'll be ok”)that additional indicators may be concerning. Although not shown,operation 1410 may also determine the positive response event 119through similar methods.

If a negative response event 115 is not determined, operation 1410proceeds to operation 1412 that may log the non-negative response event,a neutral response, and/or a positive response event 119. Where anegative response event 115 is determined (and/or where operation 1404exceeded a threshold number of attempts to determine a non-responseevent 117), operation 1414 may reference and/or retrieve previousinstances of the event of concern 116. For example, the safety profile114 and/or the log 113 of the user 100 may store each instance of thenon-response event 117 and each instance of the negative response event115 which has occurred over in a period of time (e.g., a day, a week, amonth, since implementation of the assistance coordination network). Oneor more of these previous events of concern 116 may be utilized in anevaluation of whether the adverse event 118 has occurred. In one or morealternative embodiments, process 1414 may be excluded and the event ofconcern 116 determined in process 1404 and/or 1410 may be the only eventof concern 116 evaluated. For the purposes of a specific example of theembodiment of FIG. 14, a number of previous events of concern 116 withina time period (e.g., any non-response event 117 within an eight hourperiod) may be extracted.

Operation 1406 references the safety profile 114 to determine athreshold condition to compare the number of previous events of concern116 and the recent event of concern 116 generated in operation 1404and/or 1414. For example, the safety profile 114 may specify an adverseevent 118 where: (i) any two consecutive instances of the non-responseevent 116, and/or (ii) the two instances of the negative response event115 followed by one instance of the non-response event 117 and/or (iii)any four instances of the non-response event 117 or the negativeresponse event 115 within an eight hour period. Operation 1418determines whether the event of concern 116 generated in operation 1404and/or operation 1414, plus any relevant events of concern 116 extractedfrom the log 113 and/or the safety profile 114, exceed the thresholdcondition specified in the safety profile 114. Where the thresholdcondition is not met (e.g., only one event of concern 116 occurringwithin the time period where a high number is specified), operation 1418proceeds to operation 1420. Operation 1420 may make one or moreadjustments to the safety profile 116 and/or additional data impactingthe determination of the event of concern 116 and/or the adverse event118. For example, operation 1420 may adjust the time length of the timerof operation 1404 or cause operation 1402 to occur more frequently.Additionally, in one or more embodiments operation 1420 may update thesafety profile 114 to add, remove, or modify a threshold condition uponwhere one or more events of concern 116 will be determined to be anadverse event 118.

In the specific example of the embodiment of FIG. 14, a total of fourinstances of the event of concern 116 has occurred within an eight-hourperiod. Operation 1418 compares the four instances of the event ofconcern 116 to the threshold in the safety profile 114. Upondetermination of the exceeded threshold condition, Operation 1422determines the adverse event 118 that requires an additional urgentaction beyond merely logging the event of concern 116. Specifically, theadverse event 118 determination 1422 may require execution or moreemergency actions 120. The adverse event determination process flow ofFIG. 14 is continued in FIG. 15.

FIG. 15 is an emergency action process flow illustrating one or moreemergency actions of FIG. 13 taken in response to determination of theadverse event of FIG. 14, according to one or more embodiments. Theprocess flow of FIG. 15 is continued from operation 1422 of FIG. 14.

Operation 1500 determines one or more emergency actions 120 associatedwith the adverse event 118. The safety profile 114 may specify which ofone or more emergency actions 120 should occur under what circumstances,e.g., the safety profile 114 may be referenced by operation 1500 todetermine implicated emergency actions 120. The emergency action data121 data defined and/or specifies how a specific instance of theemergency action 120 should executed. Operation 1502 determines whethera device 700 of caregiver 130 of the user 100 is to be notified as anemergency action 120. If so, operation 1504 generates an alert message122 for the caregiver 130 and that may be transmitted to the device 700of the caregiver 130 after extracting contact data (e.g., a phonenumber, an email address, a unique identifier of a mobile app userprofile or ID associated in a mobile app database). For example, thecaregiver 130 may be notified by a text message, an automated voicecall, an email, and/or a message on a mobile application (e.g., “mobileapp”). The alert message 122 may include numerous pieces of information,such as the log 113 of instances of the event of concern 116 giving riseto the adverse event 118, a GPS coordinate of a mobile device 750 of theuser 100, and/or data from connected household devices of the user 100(e.g., which lights are currently turned on in the caregivingenvironment). In either determination of operation 1502, operation 1502may proceed to a next determination of the emergency action 120, forexample operation 1506 which may make a similar determination asoperation 1502 but for notification to the emergency service 150. Theemergency action process flow then proceeds to operation 1510 whichdetermines if voice communications of the user 100 (e.g., the voicecommunication 110) should be re-routed to the emergency server 500. Ifso, operation 1512 may replicate or re-route any voice communication 110of the user 100 to the emergency server 500 and optionally increasesensitivity to an emergency speech (e.g., the emergency speech 561), forexample, by utilizing the emergency speech recognition engine 560.Although not shown, gain of a signal received may also be increasedand/or a threshold for receiving sound signals may be reduced, forexample to be able to transmit what may be a faint sound of a user 100who has fallen just on the edge of an immediate environment 102 of thedevice with a microphone (e.g., the microphone 206, the microphone 306,the microphone 706). Operation 1516 determines whether an audio and/orvideo feed should be established between the device 700 of the caregiver130 and one or more devices capable of monitoring and/or communicatingwith the user 100 (e.g., the assistance device 200, the sensing pod 300,the device 700 of the user 100, and/or other devices within thecaregiving environment such as a connected television with a web cam).If such a feed is to be established, operation 1518 establishes aconnection between devices and begins streaming the audio feed (e.g.,the audio feed 350) and/or the video feed (e.g., the video feed 352) tothe device 700 of the caregiver 130 and/or the device 700 of theemergency service 150.

Although not shown in the embodiment of FIG. 15, a two-way communicationchannel may be established in a similar way. In addition, emergencyactions 120 may depend upon an outcome of a previous emergency action120. For example, if an automated call to a phone of a first caregiver130A is unsuccessful, a second call may be placed to a second caregiver130B. Where the assistance device 200 and/or the sensing pod 300 includethe display 109 an image of the caregiver 130 may be displayed asrecorded by the device 700 of the caregiver 130. Finally, operation 1520may take any additional emergency actions 120 that may be specified andmay act as a continuation of the emergency action 120 process flow ofFIG. 15 accordant to a similar pattern shown and described. An examplemay be to turn on all of the internet-connecting lighting and turn offall music on a network-connected stereo within the caregivingenvironment to aid emergency services.

FIG. 16 illustrates additional aspects and one specific example of thedetermination of the adverse event 118, according to one or moreembodiments. Operation 1600 generates with a speaker (e.g., the speaker204, the speaker 304, the speaker 704) a verbal status query 108 to auser 100 at a predetermined time, at a random time, upon a sensing event104 in which a presence of the user 100 is sensed (e.g., by the presencesensor 202, the presence sensor 302), and/or upon a threshold inactivityof a device 700 of the user 100. The verbal status query 108 is anautomated communication 106. The predetermined time may be, for example,once every three hours, three times per day, or at 7 AM and at 5 PM(depending on a monitoring need of the user 100). In one or moreembodiments, a caretaker 130 may adjust the schedule of thepredetermined time and/or the random time. Operation 1602 initiates acountdown timer 212 set to a time length, for example 1 minutes, 20seconds, or 3 seconds. The length of time may be set for an expectedresponse of the user 100 relative to the verbal status query 108. Forexample, where the verbal status query 108 is “explain what you atetoday”, the countdown timer 212 may be set such that enough time isallowed for the user 100 to consider their response. Operation 1604determines occurrence of an event of concern 116 that is: (i) a negativeresponse event 115 that is a negative response 125 from the user 100 tothe verbal status query 108, the negative response 125 received by amicrophone (e.g., the microphone 206, the microphone 306) and/or (ii) anon-response event 117 that is a failure to receive by the microphone arecognizable response from the user 100 before expiration of thecountdown timer 212. The failure to receive the recognizable responsemay occur, for example, because no signal is received, no substantialsignal is received (e.g., no sound above a threshold ambient noise), orbecause a speech recognition software (e.g., the speech recognitionengine 460) could not determine a speech pattern or word. Operation 1606repeats the verbal status query 108 to the user 100 and optionallyincrease a volume of the verbal status query 108 from the speakerfollowing determination of the non-response event 117 of the user 100.Operation 1606 may also reduce volume of household connected devicessuch as a television or stereo.

Operation 1608, upon occurrence of the event of concern 116, transmits astatus query 107 to the device 700 of the user 100 (e.g., a text messagesent to a mobile device 750 of the user 100, a push notification set toa connected television of the user 100). Operation 1610 references asafety profile 114 specifying a condition under which one or more eventsof concern 116 define an adverse event 118 requiring execution of anemergency action 120. Operation 1612 then determines occurrence of theadverse event 118 based on the condition of the safety profile 114 andoccurrence of the event of concern 116, e.g., by comparing the one ormore events of concern 116 to the conditions in the safety profile 114.Operation 1614 determines the emergency action 120 requires anotification to at least one of a device 700 of a caregiver 130 of theuser 100 and an emergency service 150. The process flow of FIG. 16 iscontinued in the process flow of FIG. 17.

FIG. 17 illustrates additional aspects and one specific example of thedetermination of the emergency action 120, according to one or moreembodiments. Operation 1700, upon determination of the event of concern116 and/or the adverse event 118, takes any combination of the followingthree actions automatically which may be specified in one or moreinstances of the emergency action data 121. First, operation 1700 mayre-route a destination of a set of voice communications 110 of the user100 from a first voice controlled assistance service that provides ageneral assistance (e.g., provided by the automated assistance server400) to a second voice controlled assistance service having automatedemergency assistance service (e.g., provided by the emergency server500). Second, operation 1700 may re-route a voice command of the user100 from a speech recognition engine 460 providing a general speechrecognition to an emergency speech recognition engine 560 specialized inrecognizing an emergency speech 561. Third, operation 1700 may increasea recognition sensitivity to an emergency word (e.g., the emergency word1110) of the user 100. Operation 1702 generates an alert message 122 forthe emergency service 150 and/or the device 700 of the caregiver 130 ofthe user 100, the alert message 122 including a log 113 that includesthe adverse event 118 and/or the one or more events of concern 116.Finally, operation 1704, upon occurrence of the adverse event 118,establishes an audio connection (which may be the audio feed 352) and/orvideo connection (which may be the video feed 352) between (i) anassistance device 200 and/or a sensing pod 300 and (ii) the emergencyservice 130 and/or the device 700 of the caregiver 130.

In addition to initiating an emergency action 120 in monitoring the user100, the assistance coordination network of FIG. 1 also serves a numberof related functions such as streamlining use of the automatedassistance service. This encourages the user 100 to adapt to and engagewith the automated assistance server 400 and therefore the assistancedevice 200 along with the assistance coordination network and/or itselements. As a result, there may be increased opportunity to monitor,evaluate and/or protect the user 100 from the standpoint of thecaregiving organization, the caregiver 130, the emergency service 150,and/or the user 100.

FIG. 18 is an assistance application automation process flowillustrating automated interaction with an assistance application 470and an assistance action 478 of the assistance application 470conforming to a voice command protocol 474, according to one or moreembodiments. Operation 1800 selects an assistance application 470 of avoice controlled assistance service along with an assistance action 478of the assistance application 470, the assistance action 478 conformingto a voice command protocol 474 of the assistance application 470. Theassistance action 478 may be initiated by a command. For example, athesaurus application may be invoked via the automated assistance serverby saying “Launch Thesaurus.” The assistance action 478 may be a command“Synonym for”, followed by a word. Operation 1802 pre-specifies one ormore voice inputs 480 required upon initiation of the assistance action478 to execute the assistance action 478 of the assistance application470, the one or more voice inputs 480 conforming to the voice commandprotocol 474 of the assistance application 470. For example, where agiven assistance action 478 will request a voice input 480, the voiceinput 480 may be pre-recorded and/or pre-stored to be provided to theassistance application 470 without affirmative action by the user 100.

Operation 1804 optionally specifies: (i) a confirmation requirement fora verbal confirmation of the user 100 (e.g., the confirmation request1204 of FIG. 12) prior to execution of the assistance application 100and/or (ii) an input request 1200 for one or more additional inputs(e.g., one or more instances of the verbal input 1202 of FIG. 12)conforming the voice command protocol 474 before execution of theassistance application 470. Operation 1806 schedules execution of theassistance action 478 of the assistance application 470 and/or acondition for execution associated with the assistance action 478 (e.g.,the initiation condition 230). For example, the user 100 or thecaregiver 130 may specify that an assistance application 470 that ordersfood for the user 100 is scheduled to run each evening at 5 PM, or whenan inactivity of the user 100 is sensed by the assistance device 200(e.g., by the activity detection module 216).

Operation 1808 determines occurrence of the condition associated withthe assistance action 478 of the assistance application 470 of the voicecontrolled assistance service. Operation 1810 initiates a verbalconfirmation (e.g., the confirmation request 1204) of the user 100 priorto execution of the assistance application 470 and/or initiates an inputrequest 1200 for one or more additional inputs 480 conforming the voicecommand protocol 474 before execution of the assistance application 470.One instance of the condition may be occurrence of a time scheduled inoperation 1806. Operation 1812 transmits an invocation command 472 ofthe assistance application 470 and the one or more voice inputs 480required for execution of the assistance action 478. Alternatively,Operation 1812 may transmit an instruction (e.g., the initiationinstruction 232) for the assistance device 200 to initiate theassistance action 478 through the invocation command 472 of theassistance application 470 and the one or more voice inputs 480 requiredfor execution of the assistance action 478. In one or more alternativeembodiments, the assistance device 200 may provide the instruction tobegin streamlined usage of the assistance application 470 to thescheduling server 600, or the scheduling server 600 may provide theinstruction to begin streamlined usage of the assistance application 470to the assistance device 200, depending on where appropriate invocationcommands 472, initiation commands 476 and/or voice inputs 480 are storedas data.

FIG. 19 is a customizable command process flow illustrating theautomation of the assistance action 478 of FIG. 18 utilizing a customcommand word 234, according to one or more embodiments. Operation 1900defines a custom command word 234 and associates the custom command word234 with an assistance application 470 of a voice controlled assistanceservice along with an assistance action 478 of the assistanceapplication 470, the assistance action 478 conforming to a voice commandprotocol 474 of the assistance application 470. The custom command word234 for example may be “need lenses” to automatically order contactlenses to the user 100's address using existing payment details.Otherwise, the user 100 may have to remember an exact invocation command472, e.g., “Open Clear-View Eye Care Products”, and go through aworkflow within the voice command protocol 474 each time the user 100wants to order contact lenses from their preferred supplier. Operation1902 pre-specifies one or more voice inputs 480 required upon initiationof the assistance action 478 to execute the assistance action 478 of theassistance application 470, the one or more voice inputs 480 conformingto the voice command protocol 474 of the assistance application 470.

Operation 1904 specifies: (i) a confirmation requirement for a verbalconfirmation of the user 100 prior to execution of the assistanceapplication 470 and/or (ii) a requirement for an input request 1200 forone or more additional voice inputs 480 conforming the voice commandprotocol 474 before execution of the assistance application 470.Operation 1906 receives the custom command word 230 to trigger theassistance action 478 of the assistance application 470 of the voicecontrolled assistance service. Operation 1908 functions similarly tooperation 1810 of FIG. 18, and operation 1910 functions similarly tooperation 1812 of FIG. 18. Finally, operation 1912 generates a speechreply (e.g., an instance of the automated communication 106) and/or anaction executed in response to the assistance action 478 initiated andassistance application 470 invoked. The action executed may be, forexample, to control a household connected device such as dimming alightbulb or turning off a connected television.

FIG. 20 is an adverse event calibration process flow illustratingadjustment of the determination of the adverse event 118 requiring anemergency action 120 in response to the determination of an event ofconcern 116 and/or an adverse event 118, according to one or moreembodiments. Operation 1202 determines occurrence of an event of concern116 that is: (i) a negative response event 115 that is a negativeresponse 125 from the user 100 to the verbal status query 108, thenegative response 125 received by a microphone (e.g., the microphone206, the microphone 306, the microphone 706), and/or (ii) a non-responseevent 117 that is a failure to receive by the microphone a recognizableresponse from the user 100 before expiration of the countdown timer(e.g., the countdown timer 212). Operation 2002 decreases the timelength to which the countdown timer is set based on a previousoccurrence of one or more events of concerns 116 within a predeterminedtimeframe. For example, a user 100 may by default have one minute torespond before expiration of the countdown timer 212 but, after one ormore events of concern 116 the countdown timer 212 may drop to tenseconds. Operation 2004 adjusts the condition under which one or moreevents of concern 116 define the adverse event 118 requiring executionof an emergency action 120 based on the previous occurrence of one ormore events of concern 116 within the predetermined timeframe. Forexample, where a first event of concern 116 may trigger an alert message122 to a first caregiver 130 that may be a professional caregiver, asecond consecutive event of concern 116 may trigger the alert message122 to four additional instances of the caregiver 130 that may be familymembers. Finally, operation 2006 adjusts a frequency of status queries(e.g., the status query 107) based on the occurrence of the one or moreprevious events of concern 116 within the predetermined timeframe. Forexample upon occurrence of the non-response event 117, instances of thestatus query 107 may increase in frequency from once per four hours toonce per thirty minutes, possibly until a positive response event 119occurs.

The assistance coordination network may also engage in heightenedmonitoring of the caregiving environment through additional sensors thatmay detect vitals of the user, an environmental hazard to the caregivingenvironment and/or a health risk to the user 100. FIG. 21 is a personalvitals and environmental hazard emergency action process flowillustrating detection of an environmental hazard by the assistancedevice 200 of FIG. 2 and/or the sensing pod 300 of FIG. 3 and/or adversepersonal vitals from a wearable instance of the device 700 (e.g., asmartwatch that can determine a heartrate), according to one or moreembodiments. Operation 2100 may detect an environmental hazard oradverse vital, for example by receiving a signal from a particularsensor (e.g., the particulate sensor 222, the particulate sensor 322, oreven a particulate sensor 722 not shown in the embodiment of FIG. 7)and/or a chemical sensor (e.g., the chemical sensor 225, the chemicalsensor 324, or even a chemical sensor 724 not shown in the embodiment ofFIG. 7). The particulate sensor 223 may be able to sense a particulate223 such as smoke, smog, vapor, dust and additional particulate. In oneor more embodiments the particulate sensor 223 is a smoke detector thatdetects smoke. The chemical sensor 234 may be able to detect a chemical225 such as an oxidizing agent, a reducing agent, carbon dioxide, carbonmonoxide, cyanide gas, ammonia, oxygen (including lack of or overabundance of oxygen), ozone, nitrous oxides, sulfurous oxides, volatileorganic compounds, and similar hazardous molecules. The vitals mayinclude, for example, a heartbeat, a blood pressure, a temperature ofthe user 100, a blood oxygen level, a blood glucose level, and othermedical data derivable from the wearable device and/or non-wearabledevices within the caregiving environment communicatively coupled to theassistance device 200.

In the embodiment of FIG. 21, operation 2102 automatically determine anadverse event 118 upon completion of operation 2100. Operation 2104generates a warning within the caregiving environment (e.g., thecaregiving environment 800 of FIG. 8). For example the assistance device200 may broadcast to all communicatively coupled sensing pods 300through a LAN 801A a warning that could be a siren (e.g., similar to asmoke alarm siren) or may be an instance of the automated communication110 instructing the user 100 to get to safety. Operation 2106 andoperation 2108 operate substantially similarly to operation 1504 of FIG.15 and operation 1508 of FIG. 18, respectively. However, the alertmessage 122 within a context of an environmental hazard may becommunicated though more urgent means (e.g., a persistent automatedphone call or communications to all known devices 700 of the caregiver130). Similarly, operation 2110 through operation 2120 are similar tooperation 1510 through operation 1520 of FIG. 15.

FIG. 22 is a user message process flow illustrating a process by whichthe user 100 may store an away message 244 suspending a generation ofone or more verbal status queries 108 and providing notice to thecaregiver 130, according to one or more embodiments. Operation 2200stores an away message 244 of the user 100, wherein the away message 244is an audio recording 242 of the user 100. For example, the away message244 may be recorded as the audio recoding 242 directly on the assistancedevice 200, or may be recorded on the device 700 of the user 100 (e.g.,via a mobile app on a mobile device 750 of the user 100). Operation 2002then associates an expected inactivity time 240 with the away message244 of the user 100. For example the away message 244 may state that theuser 100 has gone to visit a neighbor, and the user 100 may select(e.g., on a user interface) an expected inactivity time 240 of one hour.The audio recoding 242 and/or the expected inactivity time 240 may becommunicated to a server through the network 101 (e.g., stored on theemergency server 500) or may be stored on the assistance device 200.Operation 2204 optionally adjusts the condition under which one or moreevents of concern 116 define the adverse event 118 requiring executionof the emergency action 120 based the expected inactivity time 240. Forexample, instances of the status queries 108 may continue but what mayordinarily be determined to be an event of concern 116 may not beconsidered an event of concern 116 as long as the expected inactivitytime 240 has not expired. Additionally, operation 2206 suspendsinstances of the verbal status query 208 until expiration of theexpected inactivity time 240. Operation 2208 determines expiration ofthe expected inactivity time 240. Operation 2210 generates with aspeaker a verbal status query 108 to a user 100 at expiration of theexpected inactivity time 240. In one or more embodiments the audiorecoding 242 containing the away message 244 can be delivered to thedevice 700 of the caregiver 130 upon creation, upon a check-in by thecaregiver 130 (e.g., where the caregiver 130 uses the device 700 to“ping” the assistance device 200 for the log 113), any time the user 100sets an expected inactivity time 240 of greater than one hour, and/orother similar circumstances.

FIG. 23 is an interaction of concern process flow illustrating a statusquery 108 that may be in the form of a question, a puzzle, a quiz,and/or a trivia to determine a cognitive ability of the user 100including determination of an event of concern 116 for an incorrectand/or a non-responsive answer, according to one or more embodiments.Operation 2300 generates information that may be of interest to the user100, for example a trivia related to the user 100's generation of birth(e.g., a popular culture trivia from a 1950's era). In another example,operation 2300 may invoke a history application to provide an assistanceaction 478 of describing something that occurred on this date inhistory. An affirmative response from the user 100 may be solicited toensure the user 100 heard and understood the information of interest.Operation 2302 waits a period of time, for example 24 hours, beforetaking further action. Operation 2304 generates a question, a puzzle, ora trivia question related to the information of interest, which acts asthe status query 108 that may be used to determine a status of the user100. For example, the question may be “who was the famous president youlearned about yesterday?”. In one or more alternate embodiments,operation 2304 may also begin the process flow without a need forquizzing and/or testing on information previously provided. For example,operation 2304 may test basic math or facts (e.g., “what year is it?”).Operation 2306 receives a response of the user 2306 and parses theresponse similar to operation 1408 of FIG. 14. Operation 2308 determineswhether the response received is correct and/or responsive to thequestion of operation 2304. The correct answer may be treated as apositive response event 119 and an incorrect answer (or a non-responsiveor incoherent answer) may be treated as a negative response event 115.Correct answers may be stored as data either as a text string, a number,an audio file to be compared to a response of the user 100, and/or otherdata type, and may then be associated with a particular instance of thestatus query 108. Additionally, a scoring system may be used dependingon how the degree to which the answer is correct, relevant, and/orresponse.

Upon determination of a correct and/or responsive answer, operation 2310may log the answer and/or any score, for example in the log 113 as apositive response event 119. Optionally the correct answer along withone or more other answers may be provided to the device 700 of thecaregiver 130 in operation 2312. On the other hand, where an incorrectand/or non-responsive answer is provided, operation 2314 throughoperation 2322, similar to 1414 through 1422 of FIG. 14, may be utilizedto determine whether the incorrect answer is an adverse event 118. Inone or more embodiments, events of concern 116 related to quizzes andpuzzles may be stored, tracked, and determined separately relative toevents of concern 116 related to state-based negative responses 125 ofthe user (e.g., “I don't feel well”) or a non-response event 117.

An example embodiment of the assistance coordination network will now bedescribed. Dorothy is an eighty-two-year-old widow living in a suburbanneighborhood. Dorothy has a daughter living in a nearby city, Anne, anda professional hospice social worker who checks up on Dorothy once perweek and helps her do errands such as buy groceries. Dorothy wants tocontinue to live in her home where she has lived for thirty years.However, Dorothy gets very lonely and Anne worries she does not haveenough mental stimulation or social engagement. She also has numeroushealth issues and has taken several falls in the last year. Anne and theprofessional hospice worker are concerned. Dorothy has a necklace with abutton that will notify emergency services, but she often takes it offand forgets to put it back on.

To help monitor Dorothy and provide opportunity for intuitive engagementwith technology Anne and the hospice worker install an assistance hub(e.g., the assistance device 200) in Dorothy's home. They install theassistance hub the kitchen, and three “pods” (e.g., the sensing pod 300)throughout the house, including in Dorothy's bathroom where she hasslipped before. The pods and the assistance hub communicate with oneanother through Dorothy's WiFi network. Anne and the social worker alsopreselect applications (e.g., the assistance application 470) thatDorothy can use to order food or control her television. They help herto schedule applications and set custom commands so that a single word(e.g., the custom command word 234) can trigger an entire action of anapplication even where it would normally require numerous voiceinteractions, commands or inputs.

Dorothy engages with the assistance coordination network daily. The hubasks her how she is doing at 8 AM after it senses her entering thekitchen. Around 5 PM it also asks whether she would like to order dinnerfrom one of her favorite three restaurants. Dorothy can reliablycommunicate with the hub from her kitchen and some parts of adjacentrooms when the doors are open, forming the immediate environment of thehub (e.g., the immediate environment 102 of the assistance device 200).The pods, each with their own immediate environment for communication,allow Dorothy to be in contact with the hub and additional elements ofthe assistance coordination network in almost any part of her house,forming an extended communication environment of the hub (e.g., theextended environment 103).

Dorothy has some good days and challenging days, including someoccasional memory loss. Dorothy's responses are compiled into a log(e.g., the log 113) that is sent to Anne and the social worker everyday. They have pre-selected some status queries (e.g., the status query107) that relate to Dorothy's condition, for example her frequent hippain. When Anne notices Dorothy is not responding well to queries aboutDorothy's hip generated by the hub Anne is provided the opportunity tocall to remind Dorothy to elevate her leg and take her medication.

One day Dorothy decides to do some gardening in her backyard. She hashad some vertigo earlier in the morning. When asked by the hub how shewas doing at 8 AM, Dorothy responded “I'm feeling dizzy.” The hub loggedDorothy's response as a negative response (e.g., a negative responseevent 117) but at that point the response did not arise to a levelconsidered an adverse event (e.g., the adverse event 118) based onDorothy's profile in the hub (e.g., the safety profile 114). However,the hub utilized its computer instructions to alter its procedure so itwould attempt to check in with Dorothy again in a few hours rather thanwait until 5 PM.

Dorothy gardens for an hour and then returns to the house. When enteringthe utility room to wash up she slips. She cannot get to her feet andthinks she may have broken a bone. Anne is likely busy at work and onlycalls every other day, and the social worker just visited Dorothy twodays before. Dorothy does not have her emergency necklace on because shedid not want to get it dirty when gardening.

At 11 AM the pod in the nearby living room generates a status query:“How are you feeling, Dorothy?” Dorothy hears the pod and shouts to it:“Bad. Help!” A microphone captures Dorothy's voice, and converts it toan audio signal. A voice recognition system on the hub and/or in aremote cloud computer linked to the hub through the internet (e.g., theemergency server 500) recognizes the words of the speech. The hubreferences the log and determines that Dorothy had previously generatedan event of concern earlier in the day. Dorothy's safety profileindicates that any two consecutive events of concern are an adverseevent requiring the hub take an emergency action. Dorothy's safetyprofile indicates the emergency action under these circumstance is tonotify both Anne and the social worker on their cell phones. Second,Dorothy's safety profile indicates that any two consecutive events ofconcern result in re-routing of communication from an automatedassistance server providing a general assistance to an emergency serverproviding a specialized emergency assistance.

Anne and the social worker each receive a text message specifying thatDorothy had a negative response event at 8 AM (including text ofDorothy's response to the hub) and that she had a non-response event at11 AM. The social worker is in a meeting and does not immediately seethe message. Anne however, does see it. She selects an option on amobile app of her mobile device to begin an audio-video feed from thehub and each of the pods. She cannot seem to see Dorothy but does hearsomething. Anne leaves work immediately but is several hours away. Annecalls Dorothy's phone line but there is no answer.

The hub broadcasts an audible warning through each pod that Anne and thesocial worker have been notified to let Dorothy know help may be on theway. Dorothy yells for help again. This time, the hub picks up anemergency word. The specialized emergency server (e.g., utilizing theemergency speech recognition engine 560) determines “help” with anincreased sensitivity, even though Dorothy's speech is strained due topain. The emergency server automatically calls to 9-1-1 and opens atwo-way communication between the microphone/speaker of the pod and theemergency service personnel (e.g., the emergency service 150). Emergencyservices arrive shortly after and take Dorothy to the hospital whereAnne and the social worker meet up with her.

Because of the assistance coordination network and each its components,Dorothy remains learning new things, able to easily order service, andhas access to information through an easily utilized voice interface.Through the assistance coordination network and one or more of itselements, Anne and the social worker have an enhanced capability tomonitor Dorothy for her safety and wellbeing. And the hub and/or itsassociated assistance coordination network has potentially saved Dorothyfrom serious additional injury or even death.

Although the present embodiments have been described with reference tospecific example embodiments, it will be evident that variousmodifications and changes may be made to these embodiments withoutdeparting from the broader spirit and scope of the various embodiments.For example, the various devices, engines, algorithms and modulesdescribed herein may be enabled and operated using hardware circuitry(e.g., CMOS based logic circuitry), firmware, software or anycombination of hardware, firmware, and software (e.g., embodied in anon-transitory machine-readable medium). For example, the variouselectrical structure and methods may be embodied using transistors,logic gates, and electrical circuits (e.g., application-specificintegrated (ASIC) circuitry and/or Digital Signal Processor (DSP)circuitry).

Each of the memories of the devices (e.g., the memory 203, the memory303, the memory 403, the memory 503, the memory 603) may be randomaccess memory (RAM), a storage device (e.g., an optical disk, a magneticdisk, a solid-state memory), a volatile memory, a memrister, asolid-state memory, or another type of computer-readable memory capableof storing computer bits).

In addition, it will be appreciated that the various operations,processes and methods disclosed herein may be embodied in anon-transitory machine-readable medium and/or a machine-accessiblemedium compatible with a data processing system (e.g., the serverassistance device 200, the sensing pod 300, the automated assistanceserver 400, the emergency server 500, the scheduling server 600, thedevice 700). Accordingly, the specification and drawings are to beregarded in an illustrative rather than a restrictive sense.

The structures and modules in the figures may be shown as distinct andcommunicating with only a few specific structures and not others. Thestructures may be merged with each other, may perform overlappingfunctions, and may communicate with other structures not shown to beconnected in the figures. Accordingly, the specification and/or drawingsmay be regarded in an illustrative rather than a restrictive sense.

In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. In addition, other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherembodiments are within the scope of the preceding disclosure.

What is claimed is:
 1. A system for providing assistance to a user, thesystem comprising: at least one server computer comprising at least oneprocessor and at least one memory; and an assistance device in acaregiving environment of the user, the assistance device comprising aprocessor, a memory, a speaker, and a microphone, wherein the at leastone server computer and the assistance device are configured to:receive, from a caregiver, information about a schedule of assistanceactions for interacting with the user, each assistance actioncorresponding to a predetermined time of day for implementing theassistance action, the schedule of assistance actions comprising a firstassistance action corresponding to a first predetermined time of day forinvoking the first assistance action, store, at the at least one servercomputer, the information about the schedule of assistance actions,transmit, by the at least one server computer to the assistance deviceat the first predetermined time of day, a first action instruction forimplementing the first assistance action, in response to receiving thefirst action instruction from the at least one server computer,initiate, by the assistance device, the first assistance action bygenerating an invocation command and transmitting the invocation commandto a speech processing server to obtain instructions for presentingaudio of a first input request, receive, from the speech processingserver, the instructions for presenting the audio of the first inputrequest, present, by the assistance device, the audio of the first inputrequest to the user via the speaker, receive a first voice input of theuser via the microphone, and transmit the first voice input to thespeech processing server, receive, from the speech processing server, anindication that the first voice input is a negative response event, apositive response event, or a non-response event, receive, from thecaregiver, a safety profile of the user, wherein the safety profilespecifies when a notification is to be sent to the caregiver based on atleast one of a negative response event, a positive response event, or anon-response event, and selectively notify the caregiver by causing anotification to be sent to a device of the caregiver based on the safetyprofile and the indication that the first voice input corresponds to thenegative response event, the positive response event, or thenon-response event.
 2. The system of claim 1, wherein the firstassistance action assists the user in ordering dinner.
 3. The system ofclaim 1, wherein the system comprises one or more sensing pods in thecaregiving environment of the user, wherein each sensing pod comprises amicrophone and a speaker.
 4. The system of claim 1, wherein: theassistance device comprises a presence sensor and is configured totransmit a second invocation command in response to detecting a presenceof the user with the presence sensor; and the at least one servercomputer is configured to, in response to receiving the secondinvocation command from the assistance device, implement a secondassistance action.
 5. The system of claim 1, wherein the at least oneserver computer transmits the first action instruction at the firstpredetermined time on each weekday.
 6. The system of claim 1, whereinthe schedule of assistance actions comprises a second assistance actioncorresponding to a second predetermined time of day for invoking thesecond assistance action.
 7. The system of claim 1, wherein theinformation about the schedule of assistance actions is stored at ascheduling server.
 8. The system of claim 1, wherein the at least oneserver computer comprises a scheduling server, an emergency server, andan automated assistance server.
 9. The system of claim 1, wherein theassistance device is configured to, in response to not receiving a voiceinput of the user in response to the first input request within a timeperiod: present the audio of the first input request a second time withincreased volume; or present the audio of the first input request fromanother speaker.
 10. A computer-implemented method for providingassistance to a user, the method comprising: receiving, from acaregiver, and storing information about a schedule of assistanceactions for providing assistance to the user, wherein each assistanceaction corresponds to a predetermined time of day for implementing theassistance action and conforms to a voice command protocol forpresenting input requests to the user and processing voice inputs of theuser; transmitting, by at least one server computer, a first actioninstruction at a first predetermined time of day to an assistance devicein a caregiving environment of the user to cause the assistance deviceto initiate a first assistance action by generating a first invocationcommand and transmitting the first invocation command to obtain audiofor a first input request in response to receiving the first actioninstruction; in response to receiving the first invocation command fromthe assistance device, implementing the first assistance action byinstructing the assistance device to present the audio for the firstinput request to the user, processing a first voice input of the user,and determining that the first voice input corresponds to a negativeresponse event; transmitting, by the at least one server computer, asecond action instruction at a second predetermined time of day to theassistance device to cause the assistance device to initiate a secondassistance action with a second invocation command to obtain audio for asecond input request; in response to receiving the second invocationcommand from the assistance device, implementing the second assistanceaction by instructing the assistance device to present the audio for thesecond input request to the user, processing a second voice input of theuser, and determining that the second voice input corresponds to apositive response event; transmitting, by the at least one servercomputer, a third action instruction at a third predetermined time ofday to the assistance device to cause the assistance device to initiatea third assistance action with a third invocation command to obtainaudio for a third input request; in response to receiving the thirdinvocation command from the assistance device, implementing the thirdassistance action by instructing the assistance device to present theaudio for the third input request to the user and determining that anon-response event has occurred; accessing a safety profile of the user,wherein the safety profile specifies when a notification is to be sentto the caregiver based on at least one of a negative response event, apositive response event, or a non-response event; and selectivelynotifying the caregiver by causing a notification to be sent to a deviceof the caregiver based on the safety profile and at least one of thenegative response event, the positive response event, or thenon-response event.
 11. The method of claim 10, wherein the safetyprofile of the user is updated by the caregiver.
 12. The method of claim10, wherein the first invocation command comprises speech.
 13. Themethod of claim 10, wherein determining that the non-response event hasoccurred comprises setting a timer after the third input request ispresented to the user.
 14. The method of claim 10, wherein determiningthat the negative response event has occurred comprises: determiningthat the second voice input of the user is an incorrect answer to aquestion; determining that the second voice input of the user includes aword associated with a negative state; or analyzing a tone or cadence ofthe second voice input of the user.
 15. The method of claim 10, whereinthe first assistance action presents a puzzle, a quiz, or a triviaquestion to the user.
 16. The method of claim 10, wherein the safetyprofile of the user indicates that a notification is to be sent to thecaregiver after at least one of: a specified number of negative responseevents or non-response events within a time period.
 17. One or morenon-transitory computer-readable media comprising computer executableinstructions that, when executed, cause at least one processor toperform actions comprising: receiving, from a caregiver, and storinginformation about a schedule of assistance actions for providingassistance to a user, wherein each assistance action corresponds to apredetermined time of day for implementing the assistance action andconforms to a voice command protocol for presenting input requests tothe user and processing voice inputs of the user; transmitting, by atleast one server computer, a first action instruction at a firstpredetermined time of day to an assistance device in a caregivingenvironment of the user to cause the assistance device to initiate afirst assistance action by generating a first invocation command andtransmitting the first invocation command to obtain audio for a firstinput request in response to receiving the first action instruction; inresponse to receiving the first invocation command from the assistancedevice, implementing the first assistance action by instructing theassistance device to present the audio for the first input request tothe user, processing a first voice input of the user, and determiningthat the first voice input corresponds to a negative response event;transmitting, by the at least one server computer, a second actioninstruction at a second predetermined time of day to the assistancedevice to cause the assistance device to initiate a second assistanceaction with a second invocation command to obtain audio for a secondinput request; in response to receiving the second invocation commandfrom the assistance device, implementing the second assistance action byinstructing the assistance device to present the audio for the secondinput request to the user, processing a second voice input of the user,and determining that the second voice input corresponds to a positiveresponse event; transmitting, by the at least one server computer, athird action instruction at a third predetermined time of day to theassistance device to cause the assistance device to initiate a thirdassistance action with a third invocation command to obtain audio for athird input request; in response to receiving the third invocationcommand from the assistance device, implementing the third assistanceaction by instructing the assistance device to present the audio for thethird input request to the user and determining that a non-responseevent has occurred; accessing a safety profile of the user, wherein thesafety profile specifies when a notification is to be sent to thecaregiver based on at least one of a negative response event, a positiveresponse event, or a non-response event; and selectively notifying thecaregiver by causing a notification to be sent to a device of thecaregiver based on the safety profile and at least one of the negativeresponse event, the positive response event, or the non-response event.18. The one or more non-transitory computer-readable media of claim 17,wherein the at least one server computer transmits the first actioninstruction in response to an occurrence of the first predetermined timeof day.
 19. The one or more non-transitory computer-readable media ofclaim 17, wherein the assistance device stores the first invocationcommand.
 20. The one or more non-transitory computer-readable media ofclaim 17, wherein the first invocation command comprises speech.